Re: Dynamic token kinds

[Frank Heckenbach]

Akim Demaille wrote:

> Hi Frank, Hi all,
> 
> > Le 16 déc. 2018 à 10:02, Frank Heckenbach <address@hidden> a écrit :
> > 
> > So to make it safe, we might need something like this:
> > 
> >  static inline
> >  symbol_type
> >  make_symbol (token_type type, b4_locations_if([const location_type& l, 
> > ])T&& v);
> > 
> > auto-generated for each semantic type T of any token (plus one
> > without the "v" parameter for untyped tokens) that checks (at
> > runtime) the "type" parameter against the (statically known) valid
> > token types for T.
> 
> I like this idea.  I have a draft for it in my repo, as "make-symbol".
> Please, try it and report about it.

Again, sorry for the delay (still busy), but now I tried it
(removing the "b4_parse_assert_if", see below).

It seems to work for me. The only issue I had was due to sloppiness
on my side. I'm only mentioning it in case others do the same.
Basically, I had stored tokens of one specific semantic type in a
look-up table together with tokens without semantic type (storing a
dummy value in the table for the latter), and constructed the tokens
for both in the same branch, exploiting the only case where a
mismatch is inconsequential, i.e. setting a value and not using it.
This worked before, but the stricter checks now (correctly) caught
it.

To actually allow this, you could have the typed constructors all
accept the typeless tokens as well, but I don't consider that really
necessary. Unless you want to support that for backward (bugward?)
compatibility, I'll just change my code to make two separate
make_symbol calls.

> There are a few issues:
> - make_symbol will collide if the user has a token named symbol
>   Any idea of a better name?  

To avoid such collisions, I think we have to avoid the "make_"
prefix entirely. Maybe "build_symbol"?

> Or simply make them actual constructors for `symbol_type`.

Yes, if they are (documented as) public. I think I'd prefer this as
I wouldn't have to change my code from 3.2.2.

> - In the signature of make_symbol, I've had to use int for the
>   token type, instead of the enum token_type, and then
>   to convert the int into token_type.  I don't like that, but I
>   probably don't have much of a choice.  (A template would be
>   overkill IMHO).

Why, is it because of char tokens (like '+' in your example)?
Well, it may not be too nice, but I don't really mind, especially
since the value is explictly checked here.

> - I'm not sure I should emit the assert only when parse.assert
>   was set, maybe we should always do it.  After all, if the user
>   does not want the check, he can pass -DNDEBUG.  parse.assert
>   was made to check what Bison generates, more than to check what
>   the user does.

I agree, these are different things.

Regards,
Frank