Re: cc-mode fontification feels random

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: p.stephani2@gmail.com,  ubolonton@gmail.com,  dancol@dancol.org,
>   theo@thornhill.no,  emacs-devel@gnu.org
> Date: Fri, 04 Jun 2021 14:46:12 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> a big and complex buffer would give rise to one of these big and complex
> >> JSON messages.
> > Ask Dmitry about performance problems with native JSON support, and
> > the effort we invested (a year ago?) into optimizing UTF-8 encoding of
> > strings, to squeeze every last percent of performance.
> 
> As I remember, the biggest bottleneck was parsing and allocating Lisp
> objects.

But that's exactly the problem: the packages I've seen try to solve
this on the Lisp level, and that just has got to involve consing of
Lisp objects, so there's no way around that problem with this
approach.

By contrast, fast access to buffer text is on the C level, similar to
what we do with regexp search, and doesn't require any Lisp objects as
intermediates.

The other problem with the integration of this packages into Emacs
(again, those few packages that I took a good enough look at) is that
they don't plug themselves into the JIT lock mechanism triggered by
redisplay, and instead use all kinds of hooks to put text properties
on buffer text (and turn off font-lock for that to work).  That's
another aspect of IMO poor integration into the Emacs core, probably
again because of the desire to stay away of C and the innards of the
display engine.

> Commonly, it means parsing a big JSON message even if you're
> only interested in a fraction of it (and this happens in LSP when
> e.g. some servers decide to serve up huge buckets of diagnostics
> unrelated to the current file being edited, for instance).  The json.c
> parser is faster, but ultimately borks here, too.  
> 
> My idea at the time was to develop a technique to only parse the bits of
> JSON we're interested in, which dramatically improved performance.

I think this is a separate issue.  I guess if the percentage of
"garbage" is large, then this will indeed be a win, but it must come
with some overhead (to figure out what is "garbage"), so it isn't
going to produce significant speedup with milder amounts of "garbage".

And this is only relevant if the protocol is based on JSON.

> And again, for fontification, this point is probably moot if we're going
> to integrate tree-sitter directly with direct access to the buffer
> (which just makes sense).

Only if someone does the job.