Re: Dynamic token kinds

[Frank Heckenbach]

Hans Åberg wrote:

> > On 16 Dec 2018, at 15:48, Frank Heckenbach <address@hidden> wrote:
> > 
> > Hans Åberg wrote:
> > 
> >> The idea would be that rather than returning the token value,
> >> something that does not need translation. I don't know if that
> >> helps up the static approach, though.
> > 
> > Not sure what you mean here. Please elaborate.
> 
> I couldn't see the details when I looked at it. I don't use the typed parser, 
> but might if it is safe.

The parser is type safe. This is only about an alternative way of
creating tokens by the lexer, which is also type safe when used
properly (as mine does). It's only about adding an additional safety
net.

> > I think Akim made it clear enough that he doesn't like the overhead.
> > (I don't mind so much, as I used std::variant in my implementation,
> > but I only have a few parsers to care about.)
> 
> In that case, my impression was that he thought he could do without it.

Well, he can. :)

> > One might validly say that preventing it is the job of the lexer
> > (and my lexer does so), not Bison's, just like other kinds of
> > undefined or wrong behaviour outside of the generated code, also
> > dynamic token types are a somewhat special usage anyway, so Bison
> > can just do nothing about it, that's fine.
> 
> I use the same thing, returning the token value found on a lookup
> table, but I would not want to use the typed parser if I would
> have to add translations for every possibility. The information
> about it is in Bison, therefore it should not be put on the
> writing of the lexer.

I think we agree here, and that was actually my concern when I
started this thread. I don't want to have to write a separate case
for each token kind in my lexer. Of course, we need a separate case
for each semantic type because that involves a different type in the
constructor/builder call already, but these are relatively few,
compared to token kinds, in my lexers.

> > I also suggested an approach in my previous mail with a few more
> > generated functions that help runtime checking. I'd prefer Bison to
> > add them, and then we'd have runtime checking as good as we'd have
> > with std::variant in this respect.
> 
> Maybe an option. Akim perhaps haven't used this dynamic token
> lookup.

I guess he hasn't. But I don't think we need an option. These would
just be additional functions that one can use or not.

> Those that do might prefer not risking the program to bomb.

It's not that bad actually. Again, my lexers work fine as is.
I just brought this up because Akim proposed to call the function
"unsafe_..." which I thought was too harsh and proposed
"unchecked_..." -- but by adding the checks, it would be neither
unsafe nor unchecked. :)

So to wrap this up, Akim, do you think adding the typed functions as
proposed in one of my last mails is viable?

Regards,
Frank