Bison 1.875 NOT supporting Flex 2.5.27 reentrant C scanners

[Jeannot Langlois]

Hello,


I am actually trying to generate a REENTRANT parser with Bison 1.875 (in C) 
using a REENTRANT scanner (also in C) generated by Flex 2.5.27.  I need 
everything to be in C.  The parser will be used in a multithreaded application 
(which is also in C) and will exist in multiple "instances", as different 
threads will be calling "yyparse" simultaneously.

Unfortunately, it seems to me (from all the posts and informations I've read so 
far -- please feel free to correct me if I am wrong) that parsers generated by 
Bison 1.875 are NOT calling REENTRANT C Flex scanners properly... at least yet. 
 

The problem is caused by the YYLEX definition NOT providing the "yyscanner" 
opaque structure parameter which contains the Flex scanner's state in the 
"yylex" function call occuring within "yyparse".

I've found a simple way to fix this problem by modifying the parser C source 
file AFTER it has been generated by Bison so the call to "yylex" is done 
properly by "yyparse" when the Bison parser C source file is generated with the 
proper symbol (currently YYREENTRANT_FLEX_PATCH).  

This temporary solution isn't very practical because all modications are lost 
when regenerating the parser.  I thought Bison developers would be interested 
in fixing the problem, so I am writing this little text detailing how to 
proceed with the temporary fix in the Bison-generated parser C source file.

I did not send this email to the Bison Patches mailing list because:

- this is not a problem which is specific to a grammar, 
- I am not very familiar with the Bison sources,
- this patch is for a released version of Bison (1.875),
- this function call problem is rather general,
- the file to patch (the parser C source file) varies with each different 
parser generated by Bison.

So I guess submitting a patch in this case would not be very useful.

I personnally think it would be a great idea to add support for REENTRANT Flex 
scanners in Bison by adding a command-line switch like 
"--reentrant-C-flex-scanner" which would activate the patch detailed below (the 
user knows wether he is using a REENTRANT C Flex scanner when he generates the 
Bison parser).

So here's the patch...

--------------------------------------------------------------------------------------------------------------------------------

The following procedure explains how to modify an existing C parser (generated 
with Bison 1.875) to support a reentrant C Flex 2.5.27 scanner:



1.      [parser C source file]:
        After the YYSTYPE union declaration (more precisely after the "/*Copy 
the second part of user declarations.*/" comment), 
        add the following:

        #ifdef YYREENTRANT_FLEX_PATCH
            #include "scanner.h"
        #endif

        ("scanner.h" is a header file containing the reentrant C API of the 
FLEX 2.5.27 scanner and was generated with 
        the --header-file="scanner.h" parameter when the REENTRANT C FLEX 
scanner was generated.  I know some people are 
        discussing that including this API in a Bison parser is not a good 
idea, but it seemed to me that I had no other choice).


2.      [parser C source file]:
        Modify the YYLEX definition so it looks like this:
        
        #ifdef YYREENTRANT_FLEX_PATCH
            /*new YYLEX definition which supports REENTRANT flex scanners*/
            #ifdef YYLEX_PARAM
            #define YYLEX yylex(&yylval, YYLEX_PARAM, yyscanner)
            #else
            #define YYLEX yylex(&yylval, yyscanner)
            #endif
        #else
            /*original YYLEX definition which works fine with NON-REENTRANT 
flex scanners*/
            #ifdef YYLEX_PARAM
            #define YYLEX yylex(&yylval, YYLEX_PARAM)
            #else
            #define YYLEX yylex(&yylval)
            #endif
        #endif /*YYREENTRANT_FLEX_PATCH*/
        

3.      [parser C source file]:
        At the beginning of the "yyparse" function, among the other variable 
declarations, add the following declaration:

        #ifdef YYREENTRANT_FLEX_PATCH
            yyscan_t yyscanner;
        #endif
        
        This declares an opaque scanner state object.
        

4.      [parser C source file]:
        Just before the call to YYDPRINTF((stderr, "Starting parse")) in the 
"yyparse" function, add the following:

        #ifdef YYREENTRANT_FLEX_PATCH
            yylex_init(&yyscanner);
        #endif
        
        This initializes the opaque scanner state object declared previously.
        

5.      [parser C source file]:
        Just before the "return" instruction at the end of the "yyparse" 
function, add the following:

        #ifdef YYREENTRANT_FLEX_PATCH
            yylex_destroy(yyscanner);
        #endif
        
        This destroys the opaque scanner state object declared previously.
        

6.      [probably in a Makefile or at compile time]:
        Define YYREENTRANT_FLEX_PATCH=1 (-DYYREENTRANT_FLEX_PATCH=1) when 
compiling the parser C source file with GCC.
        This will enable support to call a reentrant FLEX scanner, as explained 
in the previous steps.

        I guess this YYREENTRANT_FLEX_PATCH definition should occur when the 
user selects the "--reentrant-C-flex-scanner" 
        parameter when generating a Bison parser, as I've suggested above... (?)


--------------------------------------------------------------------------------------------------------------------------------

So that's it...
Hope this little patch helps...

Regards,

--
Jeannot Langlois
address@hidden

-- 
______________________________________________
http://www.linuxmail.org/
Now with e-mail forwarding for only US$5.95/yr

Powered by Outblaze