Re: Proposed interface changes for "parse.error custom"

[Akim Demaille]

Hi Adrian!

> Le 24 févr. 2020 à 12:03, Adrian Vogelsgesang <address@hidden> a écrit :
> 
> Hi,
> 
> * We don’t have a getter for the lookahead token. It can only be retrieved 
> through yysyntax_error_arguments. I would propose an independent getter for 
> just the lookahead token.

Why not, but where/when would you expect to use one and not the others?

> * Should yysyntax_error_arguments really be part of the public interface? I 
> see that it is used for the builtin error reporting and for testing error 
> reporting in the test cases. Still I think that the use case of 
> yysyntax_error_arguments is rather narrow and only really fits the error 
> reporting style of bison. I would prefer to not expose 
> yysyntax_error_arguments as part of the supported interface and instead 
> provide direct access to the lookahead token.

You're proposal is to expose only (i) an access to the lookahead symbol, and 
(ii) yyexpected_tokens, as opposed to exposing yysyntax_error_arguments that 
gives them both.

Why not, indeed.  I preferred the current approach because so far we don't 
expose the list of expected tokens if we don't even have a lookahead.  It's 
kind of weird to report what you wanted when you cannot even say what you did 
not like.  And it's unintuitive that this can happen, so users might be 
surprised by this.



> * yacc.c: yysyntax_error_arguments always write at index 0, even for nullptr 
> argument.

It should not, you are right!


> * Java has a different interface for yyexpected_tokens than the other 
> languages. It takes an additional yyoffset parameter. We need this, so 
> yysyntax_error_arguments can write at offset 1. If we got rid of 
> yysyntax_error_arguments, we could have the interface in sync across all 
> languages.

Right.


> * I would prefer if yyexpected_tokens would always return the number of 
> expected tokens instead of 0 in case the buffer is too small. I.e., if I pass 
> in a pre-allocated array of 5 elements but there are 10 expected tokens, I 
> would like still to get a return value of 10.

The point here is efficiency: it's useless to report the user 50 expected 
tokens, so it does make sense to say 'gimme 5 at most' and not spend time 
counting how many more there are.  That's why we "break" when we go past the 
threshold.

Yet returning 5 would not let you know where there are exactly 5, or more.  
That's why I return 0 in that case.

> That way, I can always pass in a stack-allocated 5-element array as a first 
> try and only do a heap allocation if there are more than 5 expected tokens.

I don't see where this would be useful.

If your use case in incrementally proposing possible keywords, then you want 
them all, and a max of the possible number of tokens is available statically: 
YYNTOKENS.

If your use case is error messages, you certainly want some limit, say 5, and 
then you don't need to allocate on the heap.

So I don't think one should worry about heap-allocation here in the common 
scenarios.