bug#66732: tree-sitter fontification doesn't update multi-line syntax reliably

[Stefan Monnier]

> I think most of the time the parser notifier will send ranges of limited
> size (as one or other piece of text gets recognized as e.g. a comment, or
> a raw string -- the bounds of the respective node that changed).

Yes, of course, in practice it will usually work OK.
I'm talking about what should happen ideally so that we can make an
informed decision about the code we decide to use in the end, knowing
its downsides.

>> IOW, it'd be better to mark the changed chunks with this
>> `put-text-property`, but to do it before jit&font-lock get triggered.
>> Basically, we'd like to call it from `after-change-functions`, but this
>> can be called *many* times within a single command, so we want to delay
>> it.  So we can probably just set a flag that says "there are unprocessed
>> changes" and then sprinkle calls to a "lazy update" function which
>> checks this flag before calling `treesit--font-lock-notifier` (or
>> something similar).
>
> I thought doing this lazily (inside font-lock-default-fontify-region), only
> when the region is set to be redisplayed, would be more economical than
> doing more work inside after-change-functions.

Indeed we don't want to do it directly in `after-change-functions`, but
doing it in `font-lock-default-fontify-region` is too late because the
redisplay and jit-lock have already (to some extent) decided what should
be refontified at that point.

The infrastructure does offer ways to make it work (to fix previous
incorrect assumptions about what needed to be redisplayed/refontified),
but it's best if we can avoid it.

>> One of the places where we'd need the sprinkle would be
>> `pre-redisplay-functions`.  Another one might be `syntax-ppss`?
> syntax-ppss has syntax-propertize-extend-region-functions. I haven't found
> any better suitable hooks than that.

Indeed, I don't think there's a good hook for that yet, tho maybe
`syntax-propertize-extend-region-functions` is good enough (basically
make it call `treesit--font-lock-notifier` but return nil), assuming we
have set a `syntax-propertize-function`.


        Stefan