bug#68246: 30.0.50; Add non-TS mode as extra parent of TS modes

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 5 Jan 2024 23:20:26 +0000
> Cc: casouri@gmail.com, monnier@iro.umontreal.ca, 68246@debbugs.gnu.org
> 
> On Fri, Jan 5, 2024 at 6:57 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > Of course they do!!  How else would electric-pair-mode have worked
> > > for virtually every language for more than 10 years
> >
> > forward-sexp moves forward even when there are no parentheses or
> > braces anywhere in sight.
> 
> electric-pair-mode uses scan-sexps.  Scan-sexps works perfectly to
> navigate nested mixed delimiter structures of modes that are not Lisp,
> otherwise e-p-m couldn't do it's auto-balancing job.

You are thinking about forward-sexp and scan-sexps in the situations
where you start from a parenthesis.  But scan-sexps, like
forward-sexp, handles situations where the "sexp" is not a balanced
parenthesized expression.  It uses syntax tables that in the case
where you start not from a paren yield ad-hoc results whose relation
to "sexps" is arbitrary.  Therefore, doing something similar with
results of parsing the source code will always produce arbitrary
results that are different, and make as little sense as what
forward-sexp does.

There was a discussion not long ago how to define a "sexp" in TS-based
modes so that it would make sense.  You can see there that the
conclusions are not self-evident.

But we digress.  My point in talking about sexps was that basing it on
syntax tables (like we do in traditional modes) will produce different
results than if we base them on TS parsers.  If you still disagree,
let's agree to disagree, because I already explained this twice at
least.

> > > Well the reason why e-p-m and these things work today for most ts
> > > modes is because they are also _using_ the Lisp/C parser based on
> > > syntax tables and syntax-propertize-function.
> >
> > That's because a language parser will not have any notion of a sexp,
> > so it cannot help.
> 
> As I am trying to explain, it doesn't have to be a "Lisp sexp".
> It just has to be something that scan-sexps can navigate, and
> scan-sexps works in all modes.

scan-sexps is not using the results of parsing by TS, so it doesn't
really understand the structure of the source code, and in particular
cannot provide reasonable movements by portions of expressions in at
least some languages.

Again, if you disagree, let's agree to disagree.

> I'd think that at least with a good enough grammar it's perfectly
> possible to do e.g. show-paren-mode with TreeSitter alone.

Why do you think so?  Does a TS parser produce any information about
matching parens/braces, let alone characters like <> etc?

> And the way this could work in Emacs is for TreeSitter to feed into
> scan-sexps.

I'm not sure I understand how TS could feed scan-sexps.  Did you look
at the implementation of scan-sexps?  AFAICT, there's no way to base
the code there on TS, except by providing a completely different
implementation (assuming TS parsers even provide the required
information).

> > > > I invite you to compare CC mode with c-ts-mode, and see for yourself
> > > > how the common grounds are very small.  It seems surprising at first
> > > > sight, but once you look at the code, it is very clear.
> > >
> > > And this is mainly because CC mode is, well, rather corpulent software,
> > > let's put it like that.  This is why I wrote it makes sense to start
> > > from scratch for this one.
> >
> > A discussion where you brush aside any argument that doesn't fit your
> > theory is not a useful one.
> 
> ? You write this precisely in the point where I _agree_ with you.

You "agree" for the wrong reasons.  You are, in fact, claiming that
the CC mode cannot be an example of a problem because of unrelated
reasons.  I'm saying that the reasons are related.

> > This means that the
> > base-mode hook will not see a mode that is ready for work, only its
> > beginning.
> 
> Correct.  But a major-mode doesn't have to be "ready for work" (I presume
> you mean ready for editing) for the hook to be useful.  That hook would
> be perfectly suitable for setting variables used by minor modes and other
> things. (eglot-server-program, flymake-diagnostic-functions,  
> company-backends,
> mode-line-format, etc etc)
> For turning on minor modes (eglot-ensure, company-mode, yasnippet-minor-mode,)
> For binding commands.

Stefan's patch solves these cases in a simpler manner.

> And even without the hook the mere fact that foo-mode and foo-ts-mode
> are derived from foo-base-mode according to derived-mode-p makes it
> useful.

Stefan's patch solves this in a simpler manner.

> > > >  and this various
> > > > things that you'd want to do in a mode hook are impossible in the
> > > > base-mode hook.
> > >
> > > I don't follow this part either.  Can you give an example using, say
> > > the existing ruby-base-mode.
> >
> > Again, look at the two examples I mentioned above.
> 
> I couldn't see the problem in either python.el or sh-script.el.

Search for bugs in those two files, and you will see the issues that I
had in mind.