Re: [bug-bison] Bug in string-valued terminals

[Joel E. Denny]

Hi Tom,

On Fri, 7 Jan 2011, Tom Roberts wrote:

> On 12/29/10 12/29/10 - 7:31 PM, Joel E. Denny wrote:
> > On Wed, 29 Dec 2010, Tom Roberts wrote:
> > > Ok. I decided to not use yytoknum[], as it is undocumented and also
> > > requires a
> > > funky "#define YYPRINT". Here is the code I am using inside yylex():
> > > [... some important lines omitted at beginning and end ...]
> > >                  for(int i=0; i<YYNTOKENS; ++i) {
> > >                          if(yytname[i][0] != '"') continue;
> > >                          string name(yytname[i]+1);
> > >                          name.erase(name.size()-1,1);
> > >                          for(int j=YYMAXUTOK; j>0; --j) {
> > >                                  if(yytranslate[j] == i) {
> > >                                          keyword[name] = j;
> > >                                          break;
> > 
> > Isn't your inner loop sufficient?  The body could just translate and index
> > into yytname.
> 
> Both loops are needed -- the outer one is over the string-valued tokens, and
> the inner one inverts yytranslate[] for the current string token.

Here's what I mean:

  for (int i = 0; i <= YYMAXUTOK; ++i)
    {
       char const *namep = yytname[yytranslate[i]];
       if (namep[0] == '"') 
         {
           string name = namep + 1;
           name.erase(name.size() - 1);
           keyword[name] = i;
         }
    }

For anyone just joining this thread, I don't mean to recommend either 
version of the code because...

> > In any case, YYMAXUTOK and yytranslate are not documented for users
> > either.
> 
> Yes. Perhaps they should be. Or perhaps there should be some method for
> yylex() to return a token # that does not get passed through yytranslate[]
> (e.g. a negative value, or a value with a large offset).

I'd rather document existing mechanisms than create new ones.  Of course, 
yytoknum would need to be fixed if we choose it instead of yytranslate.

> I cannot help on this

I'm thinking of emailing comp.compilers, which appears to have a much 
larger audience.