Re: treesitter local parser: huge slowdown and memory usage in a long file

[Yuan Fu]

> On May 5, 2024, at 7:04 PM, Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
> Hi Yuan,
> 
> Sorry if I'm being too pedantic here.
> 
> On 20/04/2024 05:18, Yuan Fu wrote:
> 
>> I believe I’ve found a good way to solve this problem. I pushed the changes 
>> to master.
>> Basically I added a function treesit-parser-changed-ranges that can directly 
>> return the change ranges from last reparse. This means we don’t need to use 
>> notifiers to get those change ranges anymore. Then in treesit-pre-redisplay, 
>> we reparse the primary parser and get the changed ranges from it.
>> Once we have the changed ranges, we update other non-primary parser’s 
>> ranges, but only within the changed ranges. Originally we were updating 
>> those parser’s ranges on the whole buffer, which led to the slowdown. Then 
>> we had to use some workaround to solve this. Now the workaround isn’t needed 
>> anymore.
> 
> The essence of the change (querying fewer ranges) looks good.
> 
> I'm a bit uneasy about the new function and how it's supposed to be used. 
> treesit-parser-changed-ranges returns the ranges changes during the last 
> reparse. That seems to imply that all of its callers must have the up-to-date 
> information about the state of the buffer before that reparse, and thus 
> basically follow the parser's updates through some mechanism. The 
> implementation also saves some information during every reparse, whether 
> somebody is going to call treesit-parser-changed-ranges or not.
> 
> To take our new code as an example, the only client of 
> treesit-parser-changed-ranges now is treesit--pre-redisplay, which is called 
> from syntax-propertize-extend-region-functions and pre-redisplay-functions.
> 
> Is it possible that there would occur multiple changes and reparses between 
> some firings of the above hooks? For example, some new feature might go over 
> the buffer's text with an automated multi-step transformation, calling the 
> parser (but not syntax-ppss) on each step.
> In such a scenario it seems treesit--pre-redisplay might miss intermediate 
> range updates. Would that be okay?

I think you’re right. The chance of it actually go wrong will be slim, but 
anything that’s possible to go wrong will eventually go wrong. 

The remaining question is how. I’m thinking of keeping a history of updated 
ranges, each marked with the parser timestamp. The parser timestamp is already 
there, it’s incremented every time the parser reparses. And 
treesit-parser-changed-ranges will return the timestamp along with the updated 
ranges. Then in the next iteration, the consumer can pass the last timestamp to 
treesit-parser-changed-ranges, which tells it to return all the changed ranges 
since that timestamp.

The only problem is to decide how long a history of updated ranges do we keep 
for each parser. The 100% correct approach is to maintain a separate history 
for each consumer, and never throw away old ranges until the consumer consumes 
them. But then you risk wasting memory if some consumer never consumes the 
ranges. To handle that we can add a hard limit. But then this hard limit might 
be too low for some edge case… We can make this hard limit configurable, and if 
we ever encountered a case where this hard limit is not enough and there’s no 
way around it (unlikely), we can instruct users or lisp program to increase it.

Yuan