Re: [PATCH] Fix handling of yychar manipulation in user semantic actions.

[Joel E. Denny]

Hi Akim,

On Wed, 30 Dec 2009, Akim Demaille wrote:

> Le 17 déc. 2009 à 03:29, Joel E. Denny a écrit :
> 
> > The following patch is next.  The problem it addresses is that users 
> > sometimes write semantic actions that manipulate yychar.
> 
> Wow, really?  What is a typical use of this feature?

When I've done it, it's always been an error recovery technique.

For example, I once wrote a (not well conforming) XML parser that 
recovered from mismatched start and end tags.  Its heuristic was that a 
mismatch meant that the proper end tag had simply been omitted and that 
the existing end tag was intended for an earlier start tag.  Thus, it 
assumed that this:

  <table>
    <tr>
      <td>
  </table>

really meant this:

  <table>
    <tr>
      <td>
      </td>
    </tr>
  </table>

Of course, there are some cases where this heuristic is miserably 
unhelpful, but I found it did make sense in many cases.

Anyway, my parser detected mismatches in the semantic action for the 
grammar rule whose RHS looked something like:

 START_TAG content END_TAG

If $1 and $3 did not match, the semantic action manipulated the lookahead 
in order to reinsert (END_TAG, $3, @3) back into the input, and it then 
let the grammar rule reduce as if no mismatch had occurred.  After that, 
the parser tried the same semantic action and rule for the reinserted 
END_TAG and an earlier START_TAG on the stack.

> Toying with codesearch I have found some ancient versions of G++ using 
> this, but is it really common?

I've found many projects manipulating yychar:

  
http://www.google.com/codesearch?hl=en&lr=&q=file%3A\.y+%22yychar+%3D+%22&sbtn=Search

> IMHO it is too low-level, and we should not be authorized to play with a 
> single aspect of a symbol, not with a single one at a time (especially 
> not its type!).

I agree that it does seem cleaner to have a single yyla instead of three 
different variables referring to different properties of that symbol, but 
yacc.c history is not on our side.

> BTW, YYBACKUP is not documented.

I'm afraid it is in the manual.  Even so, I agree it's a confusing 
feature, and I don't believe I've ever found reason to use it.

> And the documentation does not really 
> invite to alter yychar.

It does in at least one place, and that's my fault.  In section "GLR 
Semantic Actions" in the manual, I describe how the lookahead should not 
be modified in deferred semantic actions.  Thus, I imply that it's 
reasonable to do so in actions that are not deferred.

> > lalr1.cc does not define yychar and does not document its
> > replacement, yyla, for users.
> 
> No, indeed.  But I don't know exactly what kind of games users would 
> like to play, and which ones we want to support.

I do like the yyla concept, but I'm afraid of the backward compatibility 
implications.  It might have to be optional especially in yacc.c.