Re: Tree-sitter maturity

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: Björn Bidar <bjorn.bidar@thaodan.de>,  Philip
>  Kaludercic
>  <philipk@posteo.net>,  emacs-devel <emacs-devel@gnu.org>,  Eli Zaretskii
>  <eliz@gnu.org>,  Richard Stallman <rms@gnu.org>,  manphiz@gmail.com
> Date: Sat, 04 Jan 2025 12:39:44 -0500
> 
> The point I keep trying to make is that you can't safely update a
> foo-ts-mode tree sitter grammar without updating the corresponding
> foo-ts-mode Lisp.  They're tightly coupled.  They're not separate
> programs.  Same goes for nvim or whatever using TS grammars.
> Even distribution packagers understand the futility of consolidating
> dependencies with unstable interfaces.
> 
> When it comes to Emacs, we either 1) treat grammars as part of Emacs and
> build them with Emacs, or 2) try to take a runtime dependency on
> grammars that can be updated independently of Emacs.
> Compatibility considerations mean #2 can't work, so we're left with
> doing #1 somehow.

This is true in principle, but in practice incompatible changes in
grammar libraries are rare.  So in practice the same Lisp in
foo-ts-mode can endure quite a few changes in the tree-sitter-foo
grammar library.

> We're not talking about something like libpng, which
> could in principle be updated without Emacs having to know about the
> update.

Libraries like libpng also make incompatible ABI changes from time to
time.  I agree that they do it less frequently than tree-sitter
grammar libraries, but they still do.  And yet we don't distribute
libpng with Emacs.

> The simplest possible way to implement #1 is to just check the grammars
> into the Emacs repository and build them with Emacs using the normal
> build system.  Trying to check in hashes and download the hash-named
> grammar versions during the build and *then* build them with Emacs ---
> why bother?  Because of the hash-locking, a download-at-build-time
> scheme doesn't actually add any flexibility relative to just checking in
> the code.

This eliminates the need to keep the grammar in our repository (or
have it sub-moduled), to say nothing of the legal aspects that are
better avoided.  Also don't forget that we have at least two active
branches at any given time, and the number of grammar libraries we are
interested in is more than a handful.  So adding them to our
repository is a significant addition to the maintenance burden.

Other than that, yes, hash-locking is not much more flexible than
bundling.  I tried to tell that to people who think hash-locking is a
solution, but they still insisted.  And since they also volunteered to
maintain the DB of hashes, I don't see why I should reject that.  But
I don't think it's a good solution.

> It's just a more complicated and error-prone way of doing the
> same thing as checking in the code.  The same goes for other forms of
> downloading dependencies, e.g. via git submodules.

The difference is that the RI changes.  And that's not something to
ignore, from where I stand.