[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 00/10] Clean up the handling of the tokens
|
From: |
Akim Demaille |
|
Subject: |
[PATCH 00/10] Clean up the handling of the tokens |
|
Date: |
Sun, 12 Apr 2020 14:22:27 +0200 |
Now that the symbol kinds handling was cleaned up, it is about time to
do the same with the token kind. And to address things that have
bugged me for a very long while, and have properly annoyed others too:
- define the special tokens (YYEOF, YYUNDEF, YYERRCODE) properly
instead of via #defines.
- give them decent tags so that error messages refer to "end of file"
or "invalid token" rather than "$end" and "$undefined".
- make these built-in tags translatable if the user already translated
tokens.
- clarify some obscure messages about "user token numbers" (reported
by Frédéric Marchal, Bison's French translator).
As an example, this batch of changes (which applies to all the
skeletons) generates the following diffs on Bison's own parser:
diff --git a/src/parse-gram.h b/src/parse-gram.h
index 55912057..b40347ba 100644
--- a/src/parse-gram.h
+++ b/src/parse-gram.h
@@ -78,6 +78,8 @@ extern int gram_debug;
enum gram_tokentype
{
GRAM_EOF = 0, /* "end of file" */
+ GRAM_ERRCODE = 1, /* error */
+ GRAM_UNDEF = 2, /* "invalid token" */
STRING = 3, /* "string" */
TSTRING = 4, /* "translatable string" */
PERCENT_TOKEN = 5, /* "%token" */
diff --git a/src/parse-gram.c b/src/parse-gram.c
index 186d838e..fb74a582 100644
--- a/src/parse-gram.c
+++ b/src/parse-gram.c
@@ -106,7 +106,7 @@ enum yysymbol_kind_t
YYSYMBOL_YYEMPTY = -2,
YYSYMBOL_YYEOF = 0, /* "end of file" */
YYSYMBOL_YYERROR = 1, /* error */
- YYSYMBOL_YYUNDEF = 2, /* $undefined */
+ YYSYMBOL_YYUNDEF = 2, /* "invalid token" */
YYSYMBOL_STRING = 3, /* "string" */
YYSYMBOL_TSTRING = 4, /* "translatable string" */
YYSYMBOL_PERCENT_TOKEN = 5, /* "%token" */
@@ -643,7 +643,7 @@ yysymbol_name (yysymbol_kind_t yysymbol)
{
static const char *const yy_sname[] =
{
- N_("end of file"), "error", "$undefined", N_("string"),
+ N_("end of file"), "error", N_("invalid token"), N_("string"),
N_("translatable string"), "%token", "%nterm", "%type", "%destructor",
"%printer", "%left", "%right", "%nonassoc", "%precedence", "%prec",
"%dprec", "%merge", "%code", "%default-prec", "%define", "%defines",
@@ -670,7 +670,7 @@ yysymbol_name (yysymbol_kind_t yysymbol)
internationalizable. */
static yytype_int8 yytranslatable[] =
{
- 1, 0, 0, 1, 1, 0, 0, 0, 0, 0,
+ 0, 0, 0, 1, 1, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
@@ -900,7 +900,6 @@ enum { YYENOMEM = -2 };
#define yyerrok (yyerrstatus = 0)
#define yyclearin (yychar = YYEMPTY)
#define YYEMPTY (-2)
-#define YYEOF 0
#define YYACCEPT goto yyacceptlab
#define YYABORT goto yyabortlab
@@ -927,9 +926,6 @@ enum { YYENOMEM = -2 };
} \
while (0)
-/* Error token external number. */
-#define YYERRCODE 256
-
/* YYLLOC_DEFAULT -- Set CURRENT to span from RHS[1] to RHS[N].
If N is 0, then set CURRENT to the empty location which ends
@@ -1953,9 +1949,9 @@ yybackup:
yychar = yylex (&yylval, &yylloc);
}
- if (yychar <= YYEOF)
+ if (yychar <= GRAM_EOF)
{
- yychar = YYEOF;
+ yychar = GRAM_EOF;
yytoken = YYSYMBOL_YYEOF;
YYDPRINTF ((stderr, "Now at end of input.\n"));
}
@@ -2646,10 +2642,10 @@ yyerrlab:
/* If just tried and failed to reuse lookahead token after an
error, discard it. */
- if (yychar <= YYEOF)
+ if (yychar <= GRAM_EOF)
{
/* Return failure if at end of input. */
- if (yychar == YYEOF)
+ if (yychar == GRAM_EOF)
YYABORT;
}
else
Akim Demaille (10):
tokens: style: minor fixes
tokens: properly define the "error" token kind
tokens: define the "$undefined" token kind
tokens: properly define the YYEOF token kind
c++: remove the yy prefix from some functions
diagnostics: replace "user token number" by "token code"
skeletons: use "end of file" instead of "$end"
skeletons: make the eof token translatable if i18n is enabled
skeletons: use "invalid token" instead of "$undefined"
skeletons: clarify the tag of special tokens
TODO | 47 ++++++++-------------
build-aux/update-test | 2 +-
data/skeletons/bison.m4 | 14 +++++--
data/skeletons/c++.m4 | 17 ++++----
data/skeletons/c.m4 | 21 +++++++---
data/skeletons/glr.c | 10 ++---
data/skeletons/glr.cc | 17 ++++++--
data/skeletons/java.m4 | 15 +++----
data/skeletons/lalr1.cc | 10 ++---
data/skeletons/lalr1.java | 3 --
data/skeletons/yacc.c | 12 ++----
examples/c/bistromathic/parse.y | 3 +-
examples/c/lexcalc/lexcalc.test | 4 +-
examples/c/lexcalc/parse.y | 3 +-
examples/c/lexcalc/scan.l | 2 +-
src/output.c | 74 ++++++++++++++++++++++++---------
src/parse-gram.c | 44 +++++++++-----------
src/parse-gram.h | 2 +
src/parse-gram.y | 1 -
src/reader.c | 7 +++-
src/symtab.c | 56 ++++++++++++++++++-------
tests/actions.at | 27 ++++++++----
tests/c++.at | 2 +-
tests/calc.at | 2 +-
tests/conflicts.at | 30 ++++++-------
tests/diagnostics.at | 2 +-
tests/glr-regression.at | 4 +-
tests/input.at | 58 ++++++++++++++++----------
tests/local.at | 4 +-
tests/regression.at | 25 +++++------
tests/skeletons.at | 4 +-
31 files changed, 310 insertions(+), 212 deletions(-)
--
2.26.0
- [PATCH 00/10] Clean up the handling of the tokens,
Akim Demaille <=
- [PATCH 01/10] tokens: style: minor fixes, Akim Demaille, 2020/04/12
- [PATCH 02/10] tokens: properly define the "error" token kind, Akim Demaille, 2020/04/12
- [PATCH 03/10] tokens: define the "$undefined" token kind, Akim Demaille, 2020/04/12
- [PATCH 05/10] c++: remove the yy prefix from some functions, Akim Demaille, 2020/04/12
- [PATCH 08/10] skeletons: make the eof token translatable if i18n is enabled, Akim Demaille, 2020/04/12
- [PATCH 04/10] tokens: properly define the YYEOF token kind, Akim Demaille, 2020/04/12
- [PATCH 06/10] diagnostics: replace "user token number" by "token code", Akim Demaille, 2020/04/12
- [PATCH 09/10] skeletons: use "invalid token" instead of "$undefined", Akim Demaille, 2020/04/12
- [PATCH 10/10] skeletons: clarify the tag of special tokens, Akim Demaille, 2020/04/12
- [PATCH 07/10] skeletons: use "end of file" instead of "$end", Akim Demaille, 2020/04/12