[PATCH 0/9] Introduce and use yysymbol_code_t

[Akim Demaille]

I think I originally resisted to the idea (I believe Christian
Schoenebeck was the first to suggest it), but there are many
advantages in exposing the symbol (internal) numbers:

- custom error messages can use them to decide how to represent a
  given symbol, or a set of symbols.  See my message about custom
  error messages for examples.
  (https://lists.gnu.org/archive/html/bison-patches/2020-01/msg00000.html)

- we need something similar in uses of yyexpected_tokens.  For
  instance, currently, bistromathic's completion() reads:

    int ntokens = expected_tokens (line, tokens, YYNTOKENS);
    [...]
    for (int i = 0; i < ntokens; ++i)
      if (tokens[i] == YYTRANSLATE (TOK_VAR))
      [...]
      else if (tokens[i] == YYTRANSLATE (TOK_FUN))
      [...]
      else
      [...]

- now that's it's a compile-time expression, we can easily build
  static tables, use switch, etc.

- some users depended on the ability to get the token number from a
  symbol to write test cases for their scanners.  But Bison 3.5
  removed the table this feature depended upon (a reverse
  yytranslate).  Now they can check against the actual symbol number,
  without having pay (space and time) a conversion.  See
  https://lists.gnu.org/r/bug-bison/2020-01/msg00001.html, and
  https://lists.gnu.org/archive/html/bug-bison/2020-03/msg00015.html.

- it helps us clearly separate the internal symbol numbers from the
  external token numbers, whose difference is sometimes blurred in the
  code when values coincide (e.g. "yychar = yytoken = YYEOF").

- it allows us to get rid of ugly macros with inconsistent names such
  as YYUNDEFTOK and YYTERROR, and to group related definitions
  together.

- similarly it provides a clean access to the $accept symbol (which
  proves convenient in a current experimentation of mine with several
  %start symbols).

I have left the 'regen' commits, because it's useful to see the impact
of these changes.

These changes are only for yacc.c, but of course they will be
installed in the other skeletons too.  And I would also like to move
the lone #defines for YYEOF and YYEMPTY into yytokentype, so that we
can use strong typing for both yytoken and yychar.  And get rid of
more macros.

I'd be happy to receive comments, as usual.

Cheers!


Akim Demaille (9):
  style: comment changes about token numbers
  yacc.c: introduce an enum that defines the symbol's number
  regen
  yacc.c: use yysymbol_code_t instead of int for yytoken
  yacc.c: also define a symbol number for the empty token
  regen
  yacc.c: prefer YYSYMBOL_YYERROR to YYSYMBOL_error
  regen
  bistromathic: use symbol numbers instead of YYTRANSLATE

 TODO                            |   5 ++
 data/skeletons/bison.m4         |  31 ++++++--
 data/skeletons/c.m4             |  43 +++++++++--
 data/skeletons/yacc.c           |  53 +++++++------
 examples/c/bistromathic/parse.y |  26 ++++---
 src/parse-gram.c                | 132 +++++++++++++++++++++++++++-----
 tests/regression.at             |   2 +-
 7 files changed, 222 insertions(+), 70 deletions(-)

-- 
2.25.1