bug#66674: 30.0.50; Upstream tree-sitter and treesit disagree about fields

[Dominik Honnef]

Using tree-sitter's CLI as well as the publicly hosted playground
produce different parse trees than treesit in Emacs. Specifically, the
assignment of nodes to named fields differs.

Given the following C source:

    void main() {
      int x = // foo
        1+
        // comment
        2;
    }

treesit-explore-mode displays the following tree:

    (translation_unit
     (function_definition type: (primitive_type)
      declarator: 
       (function_declarator declarator: (identifier)
        parameters: (parameter_list ( )))
      body: 
       (compound_statement {
        (declaration type: (primitive_type)
         declarator: 
          (init_declarator declarator: (identifier) = value: (comment)
           (binary_expression left: (number_literal) operator: + right: 
(comment) (number_literal)))
         ;)
        })))

Note how in the init_declarator node, the 'value' field is a comment
node, and similarly for the 'right' field in the binary_expression node.

Running 'tree-sitter parse file.c', on the other hand, produces the
following tree:

    (translation_unit [0, 0] - [6, 0]
      (function_definition [0, 0] - [5, 1]
        type: (primitive_type [0, 0] - [0, 4])
        declarator: (function_declarator [0, 5] - [0, 11]
          declarator: (identifier [0, 5] - [0, 9])
          parameters: (parameter_list [0, 9] - [0, 11]))
        body: (compound_statement [0, 12] - [5, 1]
          (declaration [1, 2] - [4, 6]
            type: (primitive_type [1, 2] - [1, 5])
            declarator: (init_declarator [1, 6] - [4, 5]
              declarator: (identifier [1, 6] - [1, 7])
              (comment [1, 10] - [1, 16])
              value: (binary_expression [2, 4] - [4, 5]
                left: (number_literal [2, 4] - [2, 5])
                (comment [3, 4] - [3, 14])
                right: (number_literal [4, 4] - [4, 5])))))))

Here, the two comment nodes appear as unnamed nodes. IMHO the second
tree is a more useful one, as the named fields contain the semantically
important subtrees (e.g. a binary expression is made up of a left and
right subtree, not a left subtree, a right comment, and then some
unnamed subtree.)

Emacs's tree makes writing queries less convenient, as instead of being
able to refer to well-defined names, one has to rely on child indices to
account for comments.


Further mismatch arises from repeated fields and separators.

Consider the following Go source:

    package pkg
    
    var a, b, c = 1, 2, 3

treesit-explore-mode displays the following tree:

    (source_file
     (package_clause package (package_identifier))
     \n
     (var_declaration var
      (var_spec name: (identifier) name: , (identifier) value: , (identifier) =
       (expression_list (int_literal) , (int_literal) , (int_literal))))
     \n)

Here, the var_spec node has two fields named 'name' even though the
source specifies three names. Furthermore, The second 'name', as well as
'value' are set to the ',' separator between identifiers. Two of the three
identifiers aren't named.

'tree-sitter parse file.go', on the other hand, produces this more
accurate tree:

    (source_file [0, 0] - [2, 21]
      (package_clause [0, 0] - [0, 11]
        (package_identifier [0, 8] - [0, 11]))
      (var_declaration [2, 0] - [2, 21]
        (var_spec [2, 4] - [2, 21]
          name: (identifier [2, 4] - [2, 5])
          name: (identifier [2, 7] - [2, 8])
          name: (identifier [2, 10] - [2, 11])
          value: (expression_list [2, 14] - [2, 21]
            (int_literal [2, 14] - [2, 15])
            (int_literal [2, 17] - [2, 18])
            (int_literal [2, 20] - [2, 21])))))

This reproduces with 29.1 as well as 30.0.50.