should we deprecate %token-table? (was: Re: [bug-bison] Bug in string-valued terminals)

[Joel E. Denny]

On Fri, 31 Dec 2010, twlevo wrote:

> > On Thu, 30 Dec 2010 18:20:09 -0500 (EST), Joel E. Denny wrote:
> > > Oldest found doc about a option --token-table and %token-table is
> > > in gcc-1.22 or gcc-1.24 in bison.info.2 or .4 without a note how to use
> > > it.
> > Could you explain a little more about this?
> No, Not in GCC but only about GNU BISON and --token-table option
> is first found in bison-1.25 may 1996,

Ok, thanks for clarifying.

> maybe there are maillist
> archives from 1995 / 1996 with discussions about this option.

I wish.  Those years don't exist in the GNU archives, and a quick google 
search didn't find them elsewhere.

> checked the oldbison archive for token-table documentation at
> 
> https://code.google.com/p/oldbison/
> 
> There is also a patched bison version A2.3 and A2.6 (Andrew)
> which does say it is based on bison-1.22 and bison-1.24 but

So A2.3 and A2.6 have the token table functionality too?

> bison-1.25 is the first GNU official version with --token-table.
> 
> The -k, --token-table commandline option depends on a
> variable toknumflag in bison-1.25 and is implemented:
> 
> Tue Aug  1 12:30:38 EDT 1995   Wilfred J. Hansen  <address@hidden>
>         * bison.cld, getargs.c, vmsgetargs.c: Added -n, -k, and -raw switches.
>         (noparserflag, toknumflag, rawtoknumflag): New variables.

Ah, thanks, that helped.  I missed this when I searched ChangeLog before.  
Here's what I found....

yytname and yytranslate exist in 1.24.  However, there appears to be no 
directive to control whether they are generated, and they are not 
mentioned in the manual.

yytranslate is still not mentioned in the manual in 1.25, and I see no 
evidence to suggest it's meant to be associated somehow with yytname 
(other than the fact that it can be useful for yytname).

yytoknum and %token_table are both introduced in 1.25.  There, yytname and 
yytoknum are generated unconditionally if %token_table (or -k) is 
specified.  That is, there's no dependence on YYPRINT as there is in 
recent releases.  %token_table and yytname are documented in the manual.  
Unfortunately, yytoknum is not documented in the manual, but a comment in 
output.c says:

  The parser tables consist of these tables.
  Starred ones needed only for the semantic parser.
  Double starred are output only if switches are set.

  yytranslate = vector mapping yylex's token numbers into bison's token numbers.

  ** yytname = vector of string-names indexed by bison token number

  ** yytoknum = vector of yylex token numbers corresponding to entries in 
yytname

Finally, I don't see any code that accesses yytoknum in the 1.25 source. 
Thus, I'm assuming that it's not useful internally and that it's meant 
solely for Bison users.

My conclusion is that the original developers meant for users to use 
yytname with yytoknum even though they forgot to document yytoknum.  Of 
course, it's possible that they meant to allow users to choose either 
yytranslate or yytoknum, but so far there's little evidence to suggest 
that.

In any case, yytname has changed since 1.25 in a way that the original 
developers may not have intended.  The 1.25 manual says this about yytname 
entries:

  All the characters of the literal string token appear verbatim in the 
  string found in the table; even double-quote characters are not escaped.

The 2.4.3 manual (I didn't check other versions) says:

  The name in the table includes all the characters needed to represent 
  the token in Bison.  For single-character literals and literal strings, 
  this includes the surrounding quoting characters and any escape 
  sequences.

Both manuals provide examples that make the escaping differences clear.

I see two possible ways forward:

1. We document yytoknum, yytranslate, or both.  If we choose to document 
yytoknum, we make sure that it's generated when %token-table is specified 
regardless of whether YYPRINT is defined.  We assume users do not care 
about the escaped characters.  Has anyone ever complained?

2. Because neither yytoknum nor yytranslate was originally documented, we 
assume that %token-table was originally a broken feature.  Moreover, it 
may be broken further now that escaping has been introduced into the 
yytname entries.  Thus, we document %token-table and yytname as deprecated 
features and make Bison warn when %token-table is specified.  We leave 
yytoknum and yytranslate alone.  Of course, like all deprecated features, 
we won't remove %token-table until it really starts to get in the way of 
Bison maintenance.

Opinions?

> Maybe Wilfred Hansen can remember about it and how it was intended to use.

I hate to bother him with a personal email if he no longer follows Bison, 
but I'd be glad to hear his opinion if he's listening.