[bug-bison] Bug in string-valued terminals

[Tom Roberts]

I am using Mac OS X 10.5.8 (Leopard) with XCode 3.1.3; bison is:
    $ bison --version
    bison (GNU Bison) 2.3
I'm pretty sure this bug is independent of OS or environment. I have not built 
bison 2.4.3, but its bison.info still has the bug, and its code appears to.

I want to have the grammar define the keywords as literal strings, so on first 
call my yylex() will build up its list of keywords by scanning yytname[] for 
entries beginning with '"'. This avoids having to type out the entire list of 
keywords THREE times (in the grammar, in a set of %token statements, and in a 
table of keywords accessed by yylex()) -- having three lists that must be 
consistent is a major maintenance headache. My keywords do not associate and 
have no semantic values.

There are two aspects to this bug:
   A) the documentation is incomplete, tantamount to being wrong.
   B) in the output file.tab.c, the array yytoknum[] is inside a
      "#ifdef YYPRINT".
I have not built bison 2.4.3, but the bison.info in its source tarball contains 
the incomplete documentation, and yytoknum[] appears to be still inside the 
offending #ifdef.

In both bison-2.4.3/doc/bison.info and on page 84 of 
http://www.gnu.org/software/bison/manual/bison.pdf , the example code to map a 
string terminal to the return value from yylex() is incomplete -- it only gives 
a loop over yytname[], without telling the user what value to return. The loop 
variable is i, and the value that must be returned is yytoknum[i]. But 
yytoknum[] is inside that #ifdef and is not available. An alternative would be 
to search yytranslate[] for the entry equal to i, and return its index; this is 
ugly at best.

IMHO the documentation should be updated, and that #ifdef surrounding yytoknum[] 
should be removed. I have not tried this in languages other than C.

The attached sxfread.y contains two statements preceded by /*BUG ...*/ that 
illustrate my workaround. This is simple code to illustrate the bug.


Tom Roberts

/*      sxfread.y */
%{
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
int yylex(void);
void yyerror(const char *);
/*BUG This workaround makes yytoknum[] accesible. */
#define YYPRINT(A,B,C) /* need to enable yytoknum[] */
%}
%token-table
%%
input:  /* EMPTY */
        | input line
        ;
line:   "drift" '{' description '}'
        | "quadrupole" '{' description '}'
        ;
description:    /* EMPTY */
        | description word
        ;
word:   'w'
        ;
%%
int yylex(void)
{
        static char * token[] = {
                "drift","{","}",
                "quadrupole","{","w","w","}",
        };
        static int index=0;
        if(index >= sizeof(token)/sizeof(token[0])) {
                return 0;
        }
        char *t=token[index++];
        if(t[1] == '\0') {
printf("returning %d\n",t[0]);
                return t[0]; /* single-char tokens are themselves */
        }
        { char seek[259]; int i; /* this is C, not C++ */
          seek[0] = '"';
          strncpy(seek+1,t,256);
          strcat(seek,"\"");
          for(i=0; i<YYNTOKENS; ++i) {
                if(strcmp(seek,yytname[i]) == 0) {
printf("returning %d\n",yytoknum[i]);
/*BUG This is the value that must be returned: */
                        return yytoknum[i];
                }
          }
        }
printf("returning %d\n",2);
        return 2; /* error */
}

void yyerror (const char *s) 
{ 
        fprintf (stderr, "%s\n", s); 
} 

int main (void) 
{ 
        int i;
        for(i=0; i<YYNTOKENS; ++i) {
                printf("%d: %s\n",i,yytname[i]);
        }
        return yyparse (); 
}