Selective font-locking?

[JD Smith]

What is the current state of applying font-lock to only portions of a buffer? I’ve seen font-lock+ which allows adding a ‘font-lock-ignore’ property, but it redefines font-lock functions and so may not be reliable long term.  

To make this concrete, here’s a usage case.  I’m currently extending python mode to include support for multiline input.  Here is how python mode currently fontifies the text being input at a shell prompt:

1. In a post-command-hook, after every change, copies the entire input after the prompt, sans properties, to a hidden “font-lock” buffer with python-mode enabled.
2. Calls font-lock-ensure, which refontifies this entire buffer.
3. Copies all the newly updated text properties back into the shell input.

So: super inefficient.  For single lines of input, this is "fast enough".  Once you have multi-line input with hundreds of lines or more, this incurs 50-100ms latency for each and every insertion, deletion, etc.   For a good approximation of how typing with this amount of latency feels, eval the following in *scratch*:

(add-hook 'post-self-insert-hook (lambda () (sleep-for 0.1) ) nil t)

#1 is readily fixed by using an after-change-function which only updates the relevant text from the shell to the hidden buffer.  But #2 is the real killer, taking 70ms or more to completely re-fontify a decent sized block of input.  Adding region beg/end to font-lock-ensure doesn’t work; how do you know if a change occurred in a long string, for example?

But then, why bother round-tripping text out to a special-use buffer anyway, vs. just letting font-lock operate in-situ in the shell buffer itself using python-mode’s fairly simple font-lock-defaults. The only thing needed to make this work is asking font-lock to ignore all the text with ‘field of ‘output?  

It seems what would be ideal is tying font-lock-defaults to specific ‘field properties, so that only text with a given ‘field (or not matching a given ‘field) is fontified according to the matching set of font-lock rules (with no field specifier matching all text). This would make mixed multi-mode buffer fontification fairly straightforward. 

I’m sure this is simple-minded given the complexities font-lock has to solve, but there has to be a better solution than re-fontifying everything after each character is typed!