|
From: | Eric S. Raymond |
Subject: | Bison/Flex interface generation for re-entrant parsers is broken |
Date: | Thu, 9 Oct 2014 06:47:06 -0400 |
User-agent: | Mutt/1.5.21 (2010-09-15) |
The combination of the options %reentrant and %bison-bridge with %lex-param fails to generate correct code. The files included with this report demonstrate the problem and constitute the smallest possible test case for it. While the test case uses vacuous grammars, it preserves the essential features of a real case in the program cvs-fast-export. That program implements a parser for CVS master files which must be re-entrant in order to allow the most compute-intensive phase of its analysis to be parallelized. Environment: bison 3.0.2, flex 2.5.35. OS is Ubuntu 14.04 and compiler is gcc 4.8.2 but I am pretty sure the bug will reproduce elsewhere. The main bug is in bison, but this also concerns flex interfacing and exposes a flex annoyance that should be fixed. Accordingly, I'm copying it to the flex development list. To begin, drop the include files (gram.y, lex.l, yyscan.h, and Makefile) into an empty directory. Run make and observe successful code generation and compilation, as in this session transcript. make bison --defines=gram.h --output-file=gram.c -d gram.y flex --header-file=lex.h --outfile=lex.c lex.l cc -DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char *cvs)" -c -o gram.o gram.c cc -DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char *cvs)" -c -o lex.o lex.c Now comment out the CFLAGS line in Makefile and try again: make -k cc -c -o gram.o gram.c gram.c: In function ‘yyparse’: gram.c:1120:7: error: too many arguments to function ‘yylex’ yychar = yylex (&yylval, scanner, cvsfile); ^ In file included from gram.y:4:0: lex.h:326:12: note: declared here extern int yylex \ ^ make: *** [gram.o] Error 1 cc -c -o lex.o lex.c The failure occurred because, having been given this: %lex-param {yyscan_t *scanner} {char *cvsfile} bison is generating a call to yylex() correctly corresponding to it. In lex.c and lex.h, meanwhile, this piece of boilerplate code defines the actual signature of yylex(): /* Default declaration of generated scanner - a define so the user can * easily add parameters. */ #ifndef YY_DECL #define YY_DECL_IS_OURS 1 extern int yylex \ (YYSTYPE * yylval_param ,yyscan_t yyscanner); #define YY_DECL int yylex \ (YYSTYPE * yylval_param , yyscan_t yyscanner) #endif /* !YY_DECL */ Observe that while the %reentrant option of flex has correctly caused the prototype to be generated with YYSTYPE* and yyscan_t arguments, the additional char *cvsfile argument specified by %lex-params is not there. The macro definition -DYY_DECL="int yylex (YYSTYPE * yylval_param , yyscan_t yyscanner, char *cvs) works around this problem by specifying the correct prototype manually. To fufill the promises implied by its documentation, bison should generate a YY_DECL from its %lex-params and %parse-params options and include that definition in its header file where the scanner generated code will see it. I also believe that in this case flex should generate lex.h to include this: typedef void *yyscan_t; It is odd and in my opinion broken that flex generates a prototype that uses this type without including the definition before it. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
Makefile
Description: Text document
gram.y
Description: Text document
lex.l
Description: Text document
yyscan.h
Description: Text Data
[Prev in Thread] | Current Thread | [Next in Thread] |