symbol type issue with unused non-terminals

[Scheidler , Balázs]

Hi,

We, in the syslog-ng project (https://github.com/balabit/syslog-ng) have a
bison grammar file that contains a number of unused non-terminals. The
reasons for this is complicated, which I could explain if needed.

With those unused rules, bison runs into errors and generates an invalid
output.

Here's a minimal sample grammar:
```
%type <ptr> used1
%type <ptr> used2

%%
start
  : used1
  ;

used1
  : used2 { $$ = $1; }
  ;

unused
  : used2
  ;

used2
  : { $$ = (void*)0; }
  ;
```

And this is what happens when I run bison on it:
```
$ /install/bin/bison proba.y
proba.y: warning: 1 nonterminal useless in grammar [-Wother]
proba.y: warning: 1 rule useless in grammar [-Wother]
proba.y:13.1-6: warning: nonterminal useless in grammar: unused [-Wother]
 unused
 ^~~~~~
/usr/bin/m4:/install/share/bison/skeletons/yacc.c:1652: undefined macro
`b4_symbol(7, has_type)'
/usr/bin/m4:/install/share/bison/skeletons/yacc.c:1652: undefined macro
`b4_symbol(7, has_type)'
/install/share/bison/skeletons/yacc.c:1652: error: b4_symbol_if: field
has_type of 7 is not a Boolean:
/install/share/bison/skeletons/yacc.c:1652: the top level
```

The triggering condition of the issue is the presence of unused
nonterminals.

Based on my debugging I've found this root cause:

   - rules are parsed as part of the grammar, and get an associated symbol
   number
   - the RHS of rules reference terminal and non-terminal symbols using a
   symbol number. These are resolved at grammar read time and the symbol
   number is generated into the output eventually making it to m4.
   - at this point reduce_grammar() happens, this removes the unused
   non-terminal rules, causing symbols to be renumbered.
   - this makes an effort to update all symbol number references, however
   RHS of rules is not updated.
   - RHS of rules that reference "old" numbers that are higher than the
   maximum, cause those ugly m4 errors that you see above
   - At the same time, in such a case an RHS expression can easily
   reference the wrong symbol, if they got renumbered. A different
   manifestation of the same bug, where dollar actions (e.g. $1, $2, etc)
   start to use an invalid <tag> to reference the value in YYSTYPE.

To double-check, I have commented out the grammar reduction code,
recompiled bison and these errors were gone, as well as my <tag> references
became correct. This was my patch:

```
diff -urN bison-3.3.1/src/main.c bison-3.3.1-bazsi/src/main.c
--- bison-3.3.1/src/main.c      2019-01-27 14:41:42.000000000 +0000
+++ bison-3.3.1-bazsi/src/main.c        2019-01-31 18:06:31.062772501 +0000
@@ -108,7 +108,7 @@

   /* Find useless nonterminals and productions and reduce the grammar. */
   timevar_push (tv_reduce);
-  reduce_grammar ();
+//  reduce_grammar ();
   timevar_pop (tv_reduce);

   /* Record other info about the grammar.  In files derives and
@@ -208,7 +208,7 @@
   derives_free ();
   tables_free ();
   states_free ();
-  reduce_free ();
+//  reduce_free ();
   conflicts_free ();
   grammar_free ();
   output_file_names_free ();
```

This was triggered in our code-base, because macOs brew updated to bison
3.3.1 recently. If at all possible it would be great if this problem would
not spread too far (e.g. Debian). bison 3.2 still seems to work properly.

Let me know if you need any further details.
Cheers,
-- 
Bazsi