bug#56682: Fix the long lines font locking related slowdowns

[Dmitry Gutov]

On 28.07.2022 10:49, Gregory Heytings wrote:
> > > Yes, occasional mis-fontification is expected.  It's a compromise 
> > > between "no fontification" and "slow fontification".
> >
> > I wonder now if the majority of the slowdown was caused by the 
> > redisplay, whereas font-lock (which only has to run once per 
> > screenful) was actually "fast enough".
>
> Those two statements are not mutually exclusive.  The majority of the 
> slowdown was indeed caused by redisplay, but font-lock was not fast 
> enough.  Try to open a sufficiently large file (e.g. the dictionary.json 
> one) with the code on master, and type M->.

Where does one obtain dictionary.json from?

> You'll see that Emacs needs 
> about five seconds (on my laptop) to display the end of the buffer.

Does that come from the long lines, or solely from the size of the buffer?

> Now 
> compare that with the feature branch, with which the end of the buffer 
> is displayed instantaneously.  That five seconds delay is caused by 
> fontification-functions.

At some point we should accept that visiting a huge file might take some 
time (5 seconds doesn't sound terrible, depending on the context). 
Because the alternative is mis-fontifications and broken display.

> > Could you clarify what you mean by "access ... to the place where ... 
> > is defined"? "new Downloadify.Container" is highlighted by a regular 
> > regexp matcher, not some custom elisp code which has to visit the 
> > position where the identifier is defined.
>
> Sorry, I cannot be more precise, I don't have the "downloadify.js" file 
> here.  It was just a guess, based on what I saw on the screenshot, that 
> one function called by fontification-functions collects all class 
> definitions and highlights their identifiers elsewhere in the buffer 
> with a specific face.  When the buffer is narrowed, that function may 
> not see the Downloadify.Container definition (which is, I guess, placed 
> near the beginning of the file) anymore.

Here I'm attaching a version of downloadify.js we can use for comparison 
(please rename the extension from .sj to .js locally; Gmail was not 
letting it through otherwise). It's not a huge file, just about 88K.

As long as I keep my Emacs window/frame width half of the desktop, I can 
reliably reproduce the problem with the lack of highlighting for 
"Downloadify.Container" while other tokens are still highlighted.

I'm also attaching a screenshot of another problem: suddenly the bottom 
several screens of the buffer are mis-highlighted as if starting inside 
a string. That very much look like a result of breaking syntax-ppss's 
visibility of the buffer.

So the buffer scrolls quickly but looks bad.

Branch feature/long-lines-and-font-locking, revision cd41ce8c6c1079 from 
July 25. That branch is not there anymore, so let me know if I should 
re-test this with some later version of your work.

downloadify-min-example.sj
Description: JavaScript source

Screenshot from 2022-08-04 03-32-22.png
Description: PNG image