[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH] master: doc: token.prefix
From: |
Akim Demaille |
Subject: |
[PATCH] master: doc: token.prefix |
Date: |
Thu, 7 May 2009 09:13:08 +0200 |
* doc/bison.simple (Decl Summary): Document token.prefix.
(Calc++ Parser): Various fixes.
Formatting changes.
Use token.prefix.
Introduce a macro TOKEN to shorten the code and make it more
readable.
(Calc++ Scanner): Adjust.
* NEWS (Variable token.prefix): New.
---
ChangeLog | 12 ++++++
NEWS | 16 +++++++++
doc/bison.texinfo | 98 ++++++++++++++++++++++++++++++++++++++--------------
3 files changed, 99 insertions(+), 27 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index bf41280..09047f8 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,15 @@
+2009-05-11 Akim Demaille <address@hidden>
+
+ doc: token.prefix
+ * doc/bison.simple (Decl Summary): Document token.prefix.
+ (Calc++ Parser): Various fixes.
+ Formatting changes.
+ Use token.prefix.
+ Introduce a macro TOKEN to shorten the code and make it more
+ readable.
+ (Calc++ Scanner): Adjust.
+ * NEWS (Variable token.prefix): New.
+
2009-05-04 Akim Demaille <address@hidden>
bison: catch bad symbol names.
diff --git a/NEWS b/NEWS
index bbf2538..cef62ee 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,22 @@ Bison News
Also, it is possible to add code to the parser's constructors using
"%code init" and "%define init_throws".
+** Variable token.prefix
+
+ The variable token.prefix changes the way tokens are identified in
+ the generated files. This is especially useful to avoid collisions
+ with identifiers in the target language. For instance
+
+ %token FILE for ERROR
+ %define token.prefix "TOK_"
+ %%
+ start: FILE for ERROR;
+
+ will generate the definition of the symbols TOK_FILE, TOK_for, and
+ TOK_ERROR in the generated sources. In particular, the scanner must
+ use these prefixed token names, although the grammar itself still
+ uses the short names (as in the sample rule given above).
+
* Changes in version 2.5 (????-??-??):
** IELR(1) and Canonical LR(1) Support
diff --git a/doc/bison.texinfo b/doc/bison.texinfo
index b09cf84..176cc7f 100644
--- a/doc/bison.texinfo
+++ b/doc/bison.texinfo
@@ -5190,10 +5190,47 @@ is not already defined, so that the debugging
facilities are compiled.
@item Default Value: @code{false}
@end itemize
address@hidden table
@c parse.trace
+
address@hidden token.prefix
address@hidden %define token.prefix
+
address@hidden
address@hidden Languages(s): all
+
address@hidden Purpose:
+Add a prefix to the token names when generating their definition in the
+target language. For instance
+
address@hidden
+%token FILE for ERROR
+%define token.prefix "TOK_"
+%%
+start: FILE for ERROR;
address@hidden example
+
address@hidden
+generates the definition of the symbols @code{TOK_FILE}, @code{TOK_for},
+and @code{TOK_ERROR} in the generated source files. In particular, the
+scanner must use these prefixed token names, while the grammar itself
+may still use the short names (as in the sample rule given above). The
+generated informational files (@file{*.output}, @file{*.xml},
address@hidden) are not modified by this prefix. See @ref{Calc++ Parser}
+and @ref{Calc++ Scanner}, for a complete example.
+
address@hidden Accepted Values:
+Any string. Should be a valid identifier prefix in the target language,
+in other words, it should typically be an identifier itself (sequence of
+letters, underscores, and ---not at the beginning--- digits).
+
address@hidden Default Value:
+empty
address@hidden itemize
address@hidden token.prefix
+
address@hidden table
@end deffn
address@hidden %define
address@hidden ----------------------------------------------------------
%define
@deffn {Directive} %defines
Write a header file containing macro definitions for the token type
@@ -8777,13 +8814,14 @@ The code between @samp{%code @{} and @address@hidden is
output in the
@noindent
The token numbered as 0 corresponds to end of file; the following line
-allows for nicer error messages referring to ``end of file'' instead
-of ``$end''. Similarly user friendly named are provided for each
-symbol. Note that the tokens names are prefixed by @code{TOKEN_} to
-avoid name clashes.
+allows for nicer error messages referring to ``end of file'' instead of
+``$end''. Similarly user friendly names are provided for each symbol.
+To avoid name clashes in the generated files (@pxref{Calc++ Scanner}),
+prefix tokens with @code{TOK_} (@pxref{Decl Summary,, token.prefix}).
@comment file: calc++-parser.yy
@example
+%define token.prefix "TOK_"
%token END 0 "end of file"
%token ASSIGN ":="
%token <sval> IDENTIFIER "identifier"
@@ -8813,22 +8851,24 @@ The grammar itself is straightforward.
%start unit;
unit: assignments exp @{ driver.result = $2; @};
-assignments: assignments assignment @address@hidden
- | /* Nothing. */ @address@hidden;
+assignments:
+ assignments assignment @address@hidden
+| /* Nothing. */ @address@hidden;
assignment:
- "identifier" ":=" exp
+ "identifier" ":=" exp
@{ driver.variables[*$1] = $3; delete $1; @};
%left '+' '-';
%left '*' '/';
-exp: exp '+' exp @{ $$ = $1 + $3; @}
- | exp '-' exp @{ $$ = $1 - $3; @}
- | exp '*' exp @{ $$ = $1 * $3; @}
- | exp '/' exp @{ $$ = $1 / $3; @}
- | '(' exp ')' @{ $$ = $2; @}
- | "identifier" @{ $$ = driver.variables[*$1]; delete $1; @}
- | "number" @{ $$ = $1; @};
+exp:
+ exp '+' exp @{ $$ = $1 + $3; @}
+| exp '-' exp @{ $$ = $1 - $3; @}
+| exp '*' exp @{ $$ = $1 * $3; @}
+| exp '/' exp @{ $$ = $1 / $3; @}
+| '(' exp ')' @{ $$ = $2; @}
+| "identifier" @{ $$ = driver.variables[*$1]; delete $1; @}
+| "number" @{ $$ = $1; @};
%%
@end example
@@ -8869,10 +8909,10 @@ parser's to get the set of defined tokens.
# undef yywrap
# define yywrap() 1
-/* By default yylex returns int, we use token_type.
- Unfortunately yyterminate by default returns 0, which is
+/* By default yylex returns an int; we use token_type.
+ The default yyterminate implementation returns 0, which is
not of token_type. */
-#define yyterminate() return token::END
+#define yyterminate() return TOKEN(END)
address@hidden
@end example
@@ -8920,28 +8960,32 @@ preceding tokens. Comments would be treated equally.
@end example
@noindent
-The rules are simple, just note the use of the driver to report errors.
-It is convenient to use a typedef to shorten
address@hidden::calcxx_parser::token::identifier} into
address@hidden::identifier} for instance.
+The rules are simple. The driver is used to report errors. It is
+convenient to use a macro to shorten
address@hidden::calcxx_parser::token::address@hidden into
address@hidden(@var{Name})}; note the token prefix, @code{TOK_}.
@comment file: calc++-scanner.ll
@example
address@hidden
- typedef yy::calcxx_parser::token token;
+# define TOKEN(Name) \
+ yy::calcxx_parser::token::TOK_ ## Name
address@hidden
/* Convert ints to the actual type of tokens. */
[-+*/()] return yy::calcxx_parser::token_type (yytext[0]);
-":=" return token::ASSIGN;
+":=" return TOKEN(ASSIGN);
@address@hidden @{
errno = 0;
long n = strtol (yytext, NULL, 10);
if (! (INT_MIN <= n && n <= INT_MAX && errno != ERANGE))
driver.error (*yylloc, "integer is out of range");
yylval->ival = n;
- return token::NUMBER;
+ return TOKEN(NUMBER);
address@hidden
address@hidden@} @{
+ yylval->sval = new std::string (yytext);
+ return TOKEN(IDENTIFIER);
@}
address@hidden@} yylval->sval = new std::string (yytext); return
token::IDENTIFIER;
. driver.error (*yylloc, "invalid character");
%%
@end example
--
1.6.2.4
- [PATCH] master: doc: token.prefix,
Akim Demaille <=