Re: [bug-bison] Bug in string-valued terminals

[Tom Roberts]

On 12/29/10 12/29/10 - 7:31 PM, Joel E. Denny wrote:
> On Wed, 29 Dec 2010, Tom Roberts wrote:
> > Ok. I decided to not use yytoknum[], as it is undocumented and also requires a
> > funky "#define YYPRINT". Here is the code I am using inside yylex():
> > [... some important lines omitted at beginning and end ...]
> >                  for(int i=0; i<YYNTOKENS; ++i) {
> >                          if(yytname[i][0] != '"') continue;
> >                          string name(yytname[i]+1);
> >                          name.erase(name.size()-1,1);
> >                          for(int j=YYMAXUTOK; j>0; --j) {
> >                                  if(yytranslate[j] == i) {
> >                                          keyword[name] = j;
> >                                          break;
>
> Isn't your inner loop sufficient?  The body could just translate and index
> into yytname.

Both loops are needed -- the outer one is over the string-valued tokens, and the 
inner one inverts yytranslate[] for the current string token. Note that I am 
doing this only on the first call to yylex(), to fill up keyword[] (see my 
previous email for the complete C++ code snippet). From then on I just query 
keyword[] -- I decided to do the nested loops just once, and then use the much 
faster lookup in keyword[] (which is a std::map<string,int>, not an array).


> In any case, YYMAXUTOK and yytranslate are not documented for users
> either.

Yes. Perhaps they should be. Or perhaps there should be some method for yylex() 
to return a token # that does not get passed through yytranslate[] (e.g. a 
negative value, or a value with a large offset).


> Again, it would be great if someone could recall how %token-table and the
> yytname table it generates were originally intended to be used.  The
> trouble is that yytname indices are internal token numbers, but there
> seems to be no mechanism that is documented for the Bison user to
> translate internal token numbers to user token numbers (which are the
> token numbers returned by yylex).  How then is yytname useful?

I cannot help on this, as this is my first use of bison (but I used yacc 
extensively, many moons ago). BTW I'm happy with the workaround I described in 
my previous email.


Tom Roberts