Re: User Token Numbers

[Paul Eggert]

> From: Akim Demaille <address@hidden>
> Date: 08 Apr 2002 23:27:06 +0200
> 
> ISTR that the first user token number should be 257,

POSIX places two main requirements:

  (A) yacc-assigned token numbers must be greater than 256.

  (B) yacc-assigned token numbers must be distinct from token numbers
      already assigned.

Starting with 257 suffices for (A), though it's not necessary (you
could start with 258, say).

However, starting with 257 does not suffice for (B), since the user
can assign token number 257 by writing `%token FOO 257' first thing,
before any other part of the grammar that assigns token numbers.

> now that I have $undefined moved, I must put it past the user
> tokens, right?

Not necessarily.  You could make it 257 in the usual case, as
described below.

> 256 is `error'

Yes, unless the user says `%token error <some other number>'.

> 257 is the last user token

Not necessarily; the first yacc-assigned token number could be 258.
Bison could treat $undefined somewhat like error: $undefined's number
could be 257 unless the user says `%token $undefined N' where N !=
257.  However, it's a bit different from `error' because it's valid to
say `%token someothertoken 257' without mentioning `$undefined',
whereas it's not valid to say `%token someothertoken 256' without a
preceding `%token error N' where N != 256.

We have the following sets of tokens:

 (A) the error token, but only if the user has not assigned its number
 (B) other builtin tokens like $undefined
 (C) unibyte literals
 (D) token numbers assigned explicitly by the user with `%token FOO N'
 (E) tokens not assigned yet

(A) is the empty set if the user explicitly assigned the error token's number.

Assuming Bison is assigning token numbers at the end, then the
straightforward algorithm, as I see it, is:

 (1) Check that the token numbers declared in (A), (C) and (D) do not
     collide with each other.

 (2) Assign numbers starting with 257 to the tokens in categories (B)
     and (E), in each case taking care to avoid collisions with all
     previous assignments.