Re: treesitter local parser: huge slowdown and memory usage in a long file

[Dmitry Gutov]

On 09/05/2024 03:16, Yuan Fu wrote:

> > Is it possible that there would occur multiple changes and reparses between 
> > some firings of the above hooks? For example, some new feature might go over 
> > the buffer's text with an automated multi-step transformation, calling the 
> > parser (but not syntax-ppss) on each step.
> > In such a scenario it seems treesit--pre-redisplay might miss intermediate 
> > range updates. Would that be okay?
>
> I think you’re right. The chance of it actually go wrong will be slim, but 
> anything that’s possible to go wrong will eventually go wrong.

Thanks for confirming the concern.

> The remaining question is how. I’m thinking of keeping a history of updated 
> ranges, each marked with the parser timestamp. The parser timestamp is already 
> there, it’s incremented every time the parser reparses. And 
> treesit-parser-changed-ranges will return the timestamp along with the updated 
> ranges. Then in the next iteration, the consumer can pass the last timestamp to 
> treesit-parser-changed-ranges, which tells it to return all the changed ranges 
> since that timestamp.
>
> The only problem is to decide how long a history of updated ranges do we keep 
> for each parser. The 100% correct approach is to maintain a separate history 
> for each consumer, and never throw away old ranges until the consumer consumes 
> them. But then you risk wasting memory if some consumer never consumes the 
> ranges. To handle that we can add a hard limit. But then this hard limit might 
> be too low for some edge case… We can make this hard limit configurable, and if 
> we ever encountered a case where this hard limit is not enough and there’s no 
> way around it (unlikely), we can instruct users or lisp program to increase it.

That could work. Although it's hard for me to imagine how far back the 
history would have to be stored, and would that have any practical 
consequences for Emacs's memory use. Maybe not.

The approach I was thinking of is in different direction: we take a step 
back, remove (or stop using at least) the new function, and go back to 
the idea of subscribing to parsers' after-change notifiers. The 
improvement in commit f62c1b4cd00 seems to stem from relying foremost on 
changes ranges in the primary parser. Okay - we re-add the listener for 
the primary parser only.

This listener would be specific for a particular consumer. In our case, 
we'd have a listener which would populate - and then update - the 
variable used by treesit--pre-redisplay. That variable would store the 
"up to date" list of updated ranges. The listener, on every call, would 
"merge" its current value one with the new list of ranges (*). 
treesit--pre-redisplay would use the data in that data structure instead 
of calling treesit-parser-changed-ranges, and set the value to nil to 
"reset" it for the next update.

(*) So real "merging" would only need to be performed when listener 
fires 2+ times between the two adjacent treesit--pre-redisplay calls. 
Otherwise the current value is nil, so the the new list is simply 
assigned to the variable. Anyway, the merging logic seems to be the 
trickiest part in this scheme (managing and interpreting offsets), but 
it should be very similar in both approaches.