Bug in bison-1.29 with comment handling

[Keith Browne]

I discovered a problem with bison-1.29 when using it to build lclint
(http://lclint.cs.virginia.edu/).  lclint is an enhanced lint tool
which can perform extended source code checking through the use of
markup tokens encapsulated in C comments, like this:

  /address@hidden@*/ arrayQualNode array;

Not surprisingly, lclint's own source code is marked up in this
manner.  In particular, a couple of the grammar files used to build
lclint have declarations like the one above.

When building lclint with bison-1.29, I found bison was placing an
extra / in front of C comments in its output--thus generating a
C++-style comment to end of line.  This would be harmless for C
comments which extend to end-of-line, but it breaks code where the
comment precedes other code on a line, or where a comment extends over
multiple lines.

I've modified the mfcalc grammar provided in the bison manual to
illustrate this problem.  Here's the modified grammar, with extra
comments in the union specification:

-----------------------------------------------------------------
%{
#include <math.h>  /* For math functions, cos(), sin(), etc. */
#include "calc.h"  /* Contains definition of `symrec'        */
%}
%union {
  /address@hidden@*/double     val;  /* For returning numbers.                  
 */
  /*
   * multi-line comment
   */
  /address@hidden@*/symrec  *tptr;   /* For returning symbol-table pointers     
 */
}

%token <val>  NUM        /* Simple double precision number   */
%token <tptr> VAR FNCT   /* Variable and Function            */
%type  <val>  exp

%right '='
%left '-' '+'
%left '*' '/'
%left NEG     /* Negation--unary minus */
%right '^'    /* Exponentiation        */

/* Grammar follows */

%%
input:   /* empty */
        | input line
;

line:
          '\n'
        | exp '\n'   { printf ("\t%.10g\n", $1); }
        | error '\n' { yyerrok;                  }
;

exp:      NUM                { $$ = $1;                         }
        | VAR                { $$ = $1->value.var;              }
        | VAR '=' exp        { $$ = $3; $1->value.var = $3;     }
        | FNCT '(' exp ')'   { $$ = (*($1->value.fnctptr))($3); }
        | exp '+' exp        { $$ = $1 + $3;                    }
        | exp '-' exp        { $$ = $1 - $3;                    }
        | exp '*' exp        { $$ = $1 * $3;                    }
        | exp '/' exp        { $$ = $1 / $3;                    }
        | '-' exp  %prec NEG { $$ = -$2;                        }
        | exp '^' exp        { $$ = pow ($1, $3);               }
        | '(' exp ')'        { $$ = $2;                         }
;
/* End of grammar */
%%
-----------------------------------------------------------------

I'll excerpt the relevant output from the beginning of the
mfcalc.tab.c file generated by bison 1.25:

bash-2.04$ /usr/local/bin/bison --version
GNU Bison version 1.25

-----------------------------------------------------------------

/*  A Bison parser, made from mfcalc.y
 by  GNU Bison version 1.25
  */

#define YYBISON 1  /* Identify Bison output.  */

#define NUM     258
#define VAR     259
#define FNCT    260
#define NEG     261

#line 1 "mfcalc.y"

#include <math.h>  /* For math functions, cos(), sin(), etc. */
#include "calc.h"  /* Contains definition of `symrec'        */

#line 5 "mfcalc.y"
typedef union {
  /address@hidden@*/double     val;  /* For returning numbers.                  
 */
  /*
   * multi-line comment
   */
  /address@hidden@*/symrec  *tptr;   /* For returning symbol-table pointers     
 */
} YYSTYPE;
...
-----------------------------------------------------------------

The output continues, but note that the comments have been copied
correctly as C comments.  Here's the output from bison-1.29:

bash-2.04$ /usr/ubik/bin/bison --version
bison (GNU Bison) 1.29

Copyright 1984, 1986, 1989, 1992, 2000, 2001 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

-----------------------------------------------------------------

/* A Bison parser, made from mfcalc.y
   by GNU bison 1.29.  */

#define YYBISON 1  /* Identify Bison output.  */

# define        NUM     257
# define        VAR     258
# define        FNCT    259
# define        NEG     260

#line 1 "mfcalc.y"

#include <math.h>  /* For math functions, cos(), sin(), etc. */
#include "calc.h"  /* Contains definition of `symrec'        */

#line 5 "mfcalc.y"
typedef union {
  //address@hidden@*/double     val;  //* For returning numbers.                
   */
  //*
   * multi-line comment
   */
  //address@hidden@*/symrec  *tptr;   //* For returning symbol-table pointers   
   */
} YYSTYPE;
#include <stdio.h>
...
-----------------------------------------------------------------

Here, the comments which appear before the type specifiers in the
union have been converted to C++ // comments, so gcc won't see the
type declarations at all.  The multi-line comment is similarly broken.

The above output was generated on a Solaris 2.5.1 machine.  I'm also
able to get the expected behaviour (C comments instead of C++) with
bison-1.27 on Solaris, and with bison-1.28 on NetBSD/x86 1.5--I wasn't
able to compile bison-1.28 on Solaris.

This problem probably doesn't show up in most .y files, since most
people won't be using multiline comments or comments which precede
other code on a line.

Keith D. Browne
Programmer, Legato Systems Canada Inc.
address@hidden