Re: [SPAM UNSURE] Re: Reliable after-change-functions (via: Using incremental parsing in Emacs)

[Stephen Leake]

Tuấn-Anh Nguyễn <address@hidden> writes:

>> > My suggestion is first to figure out how to do this stuff efficiently
>> > from within Emacs itself, as if the module interface were not part of
>> > the equation. We can add that aspect back later.
>>
>> There are two times the wisi code that wraps the parser needs access to
>> the buffer; first to copy the text, second to add text properties
>> (faces, indent values, navigation markers). There are usually many text
>> properties output by each parse.
>>
>> The positions and values of the text properties are computed by
>> functions that run after the complete syntax tree has been produced. In
>> wisi, those functions are added directly in the grammar source file
>> (where they are called "post-parse grammar actions"). In tree-sitter, I
>> assume they are called from some mode-author-written code that traverses
>> the syntax tree (wisi provides that internally). Except I see below that
>> the emacs tree-sitter package stores the syntax tree in the buffer.
>>
>
> The preferred approach with tree-sitter is querying:
> https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries

For access to the syntax tree, yes. There must be code somewhere that
computes face, indent and navigation (and code completion, ...). That
code will build on top of the syntax tree access; it could be in Rust
(in the module) or in elisp (using the module functions).

Or in C linked directly into Emacs, as Eli suggests. But I don't think
he meant that as an actual implementation approach, just as a design
approach.

The wisi Ada code that computes the text properties accesses the syntax
tree more directly, but that's just an implementation detail.

I think it makes sense at this point to try to merge wisi and
emacs-tree-sitter. There are several approaches:

1. rewrite the wisi grammar actions in elisp, using the
emacs-tree-sitter module functions to access the syntax tree.

2. rewrite the wisi grammar actions in Rust, using Rust functions to
access the syntax tree

3. rewrite the emacs-tree-sitter module in Ada, using an Ada binding to
the Tree-Sitter C API. Then the Emacs module would provide the current
wisi Ada code, modified to work with a Tree-Sitter parser.

4. There would also be value in doing an independent design and
implementation of code to compute face, indent and navigation using the
tree-sitter syntax tree; there might be a better approach than what wisi
does.

1 is probably the quickest path to getting something working, but 2 or 3
will probably provide faster execution time. Ideally we'd do all three
(or four) and get some good metrics.

After doing one of the above, we must still write the calls to the
grammar actions for each language of interest. In wisi, this is done by
adding grammar actions to the grammar source code; for example, here is
the indent action for the Ada 'if then end if' statement:

if_statement
  : IF expression_opt THEN sequence_of_statements_opt END IF SEMICOLON
    %((wisi-indent-action [nil [(wisi-hanging% ada-indent-broken (* 2 
ada-indent-broken))
                              ada-indent-broken]
                             nil
                             [ada-indent ada-indent] nil nil nil]))%

There is one lisp form for each token in the grammar production.

IF is not indented by this action; it is indented by the enclosing Ada
statement.

The conditional expression is indented by wisi-hanging; comments within
the expression (assuming it is multi-line) are indented by
ada-indent-broken. wisi-hanging takes care of indenting the second line
in a long expression.

THEN, END IF SEMICOLON are not indented. The statements in the true
branch are indented by ada-indent.

If ada-indent is 3, ada-broken-indent 2, this produces:

     if a or
        b
        -- a comment
    then
       statement_1;
       statement_2;
       --  another comment
    end if;

In the upstream development repository for the wisi package
(https://savannah.nongnu.org/projects/ada-mode/), there is a user guide
to the grammar actions. I can provide an html or info version on
request. (or get my act together and do another wisi/ada-mode release).

In Tree-Sitter, the calls to the grammar actions are written in code
that traverses the syntax tree; this would be a higher level elisp or
Rust function, for 1 and 2 above.

-- 
-- Stephe