bug#65451: 30.0.50; `after-change-functions' are not triggered in the same order the changes are made

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Yuan Fu <casouri@gmail.com>, 65451@debbugs.gnu.org
> Date: Tue, 22 Aug 2023 13:41:17 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Then, I'd like to point back to the previous discussion where I asked to
> >> expose to Elisp information about buffer changes available to
> >> tree-sitter.
> >> https://yhetil.org/emacs-devel/83tu8jq2vl.fsf@gnu.org/
> >
> > I don't want to do that, sorry.  Not without a good understanding of
> > what exactly do you need from that and in what way.  If we will expose
> > anything, it will have to be the minimum possible exposure, not the
> > maximum, so I would like to understand this very well before I agree
> > to any change in this direction.
> 
> Org wants to do the same thing tree-sitter does - keep parsed AST in
> sync with buffer modifications without having to re-parse the whole
> buffer. So, we basically need the same information tree-sitter needs -
> the sequence of buffer text changes, in their order.

We don't expose the data you want to tree-sitter in Lisp.  What is
exposed to Lisp are the parser and parse-tree objects that we build
(in C) based on tree-sitter parsing results.  When the buffer is
modified, the information about the modifications is used internally
by Emacs, in C code, to find and update the relevant parsers, and for
that we call the tree-sitter functions involved in this process.  See
the function treesit_record_change which does that, and which is
called from C when buffer text changes in a way relevant to treesit.el
functionalities.  (Note that some changes of buffer text are not
visible even to tree-sitter, because we decided they are not relevant,
for now.)

> Note that the markers discussed in the thread I linked are not
> sufficient. When editing near AST node boundaries, even if the
> boundaries are represented by markers, we have to re-parse the AST
> around to account for the possible structural changes. So, information
> about buffer edits is still required.

If tracking markers is not enough, then I wonder how the information
from the lower levels, which is basically the same but noisier, will
be able to help you.

> >> In fact, I am not sure if tree-sitter will behave correctly if it is
> >> signaled changes in incorrect order.
> >
> > I will defer to Yuan, but tree-sitter doesn't use these hooks, we call
> > its functions directly from insdel.c where needed.  This makes sense
> > for a library to which we link and whose interface code we control,
> > but giving such access to Lisp (and Org on top of that) is out of the
> > question.  We don't even give such access to modules.
> 
> I hope that we can solve this issue one way or another. This currently
> breaks the very core functionality of Org. Every part of Org relies on
> it to obtain reasonable performance. Prior to using cache, we had orders
> of magnitude slowdowns.

If you can arrange your design such that Lisp sees only AST-specific
objects affected by the modifications in buffer text, then I believe
we will have a good chance of finding a satisfactory solution.  If
that requires to have some of your code in C (preferably, generalized
to some extent), then so be it.

You see, I think the buffer-change hooks we have are already too much:
Lisp programs abuse them all the time (you can see a good example in
the bug which I mentioned up-thread, and which led to the change you
are now complaining about).  Doing more of that is not very wise, to
say the least.

Moreover, I think the solution you think you want you actually _don't_
want, because it will overwhelm you with changes that are not relevant
to your purposes.  You can see a clear evidence to that in the fact
that treesit_record_change is called only in several strategical
places, not everywhere where we change buffer text, and not at the
lowest level of such changes.  There's a reason to that.