Re: Bison/flex compatibility revisited

[Bruce Lilly]

John wrote:
> > YYFARGS0 YYFARGS1 YYFARGS2 YYFARGS3 YYLMAX YYLTYPE
> > YYLTYPE_IS_DECLARED YY_BREAK YY_BUFFER_EOF_PENDING YY_BUFFER_NEW
> > YY_BUFFER_NORMAL etc.
>
>
> The above list of macros is misleading. The macros are defined in the
> header, then #undef'd at the end of the header. Only a select few
> remain #undef'd.

Let's consider several scenarios and the implications:

1. Flex header not generated or not included by parser

  Compiling the parser yields warnings such as

    y.tab.c: In function `yyparse':
    y.tab.c:4321: warning: implicit declaration of function `yylex'

  Thus there is no checking, and if the flex options are not compatible
  with the bison options, there may be mysterious run-time errors.
  OTOH there's no name-space clash to worry about.

2. Flex header generated and included by parser, but w/o the
  magic #define YY_HEADER_NO_UNDEFS (or equivalent) before the
  inclusion.

  Same compilation warnings as above, same consequences.  However,
  there is the potential for *some* name-space clashes, as *some*
  of the flex-defined names are visible (e.g. YY_FLEX_SUBMINOR_VERSION,
  which is a recent addition that probably was overlooked; there
  are others).

  Note that although it does not appear to have happened yet in
  practice, there is the possibility that a Bison YY or yy
  name might cause strange things to happen with the included
  flex header (the obvious example would be if Bison were to
  define YY_HEADER_NO_UNDEFS as a macro taking one or more
  arguments).

In either of the above two cases, the user could manually provide
a declaration in the parser file, but of course that would not be
correct if certain flex options were changed, since the actual
definition of yylex changes (name, number and type of arguments)
with the options specified.  Or one could slice, dice, and
fricassee the flex-generated header with sed or something similar,
but changes to flex will require changes to the slicing and dicing
and perhaps the fricasseeing as well.  Or one could process the
flex-generated .c file to extract the macro and prototype (but
changes to the flex skeleton or to indent may break that). The
only way (currently) to make the yylex declaration that is in
the flex-generated header visible (and the macro renaming it if
flex's prefix option is used) is to define YY_HEADER_NO_UNDEFS
prior to the inclusion, and that's the next scenario

3. Flex header generated and included, YY_HEADER_NO_UNDEFS defined
   to make yylex macro and prototype visible

  Compilation warnings go away (iff flex and bison options are
  compatible; errors are generated if they are not, and that is
  of course the whole point of prototypes in the first place).
  Basically all of the flex yy and YY prefixed names are exposed
  (as well as several others).  That's what led to the earlier
  clash on yyout with Bison 1.75b, and what led to the clash
  on yylineno with Bison 1.875.

One could of course follow the inclusion directive in the parser by
a bunch of explicit #undef directives, but the list of names to
#undef is quite long, and may change with changes in flex.  Or one
could process the flex-generated header, with all of the caveats
mentioned above regarding that approach.

4. For whatever reason, the programmer doesn't care about the
   yylex macro or prototype, but needs start condition definitions

  One could of course process the flex-generated .c file to extract
  them into a header file as an alternative to using the flex-
  generated header, with the caveats mentioned above regarding
  that approach. It also requires some trickery, as there is
  normally no type of signature that identifies the start
  condition macro definitions.  If the flex-generated header
  file is included by the parser, *both* YY_HEADER_NO_UNDEFS
  and YY_HEADER_EXPORT_START_CONDITIONS have to be defined,
  and all of the characteristics of #3 above apply.

Anyway, that's the situation as I see it.

I think some kind of arrangement between the flex and bison
developers will need to be an essential part of resolving
these issues for the common case of a flex lexical analyzer
coupled with a bison-generated parser. As Paul correctly
points out, there are issues if a different compiler compiler
or lexical analyzer generator are used (but I don't think
an exactly analagous situation arises because (a) most
other lexical analyzer generators either don't produce a
header file or don't define nearly as many yy and/or YY
prefixed names, and (b) most other lexical analyzer
generators and compiler compilers don't have as many options
that affect the calling convention for yylex). From Paul's
decription, it certainly also sounds like one of the POSIX
committees needs to do some work to remove the contradictions
in the standard.

Aside from that, from my perspective it would be nice to
have some degree of control over what is exposed by the
flex-generated header, rather than the current basically
all-or-nothing choice.  Specifically, I'd want the yylex
macro and declaration visible essentially every time I
use flex, separately from all of the other stuff, and
I often need to be able to use start condition macro
definitions (either separately or in combination with
the yylex stuff).  Most of the other things I don't
really have need for.  Ideally if they're not needed
they wouldn't be in the header at all, since their presence
does lead to the potential for clashes, but apparently
that's difficult to implement, so #undef'ing them at
the end is OK -- provided I can still get the yylex
stuff and/or start conditions without the whole kit and
kaboodle, and provided that there is some kind of
clash-avoidance arrangement as mentioned in the previous
paragraph.

Best regards,
  Bruce Lilly