outdated m4sugar

[Eric Blake]

Right now, the latest bison sources are incompatible with the latest m4 
sources.  When I built bison with the latest m4.git (will become m4 1.6), the 
change from LIFO to FIFO m4wrap (done in order to make m4 obey POSIX) trips up 
a fresh build of bison:

$ m4 --version | head -n1
m4 (GNU M4) 1.5.89a.15-21859
$ sed -n '1,4 { s/<.*>/<...>/; p }' ChangeLog 
2008-05-27  Joel E. Denny  <...>

        * NEWS (2.3b+): New section, empty for now.
        * configure.ac (AC_INIT): 2.3b -> 2.3b+.
$ make
...
make[2]: Entering directory `/home/eblake/bison/examples/calc++'
../../tests/bison -d -ra -S lalr1.cc -o ./calc++-parser.cc \
          ./calc++-parser.yy
stdin:1: error: m4_init: unbalanced m4_divert_push:
stdin:1: m4_divert_push: KILL
stdin:1: the top level
/home/eblake/bison/src/bison: subsidiary program `/usr/local/bin/m4' failed 
(exit status 1)

I tracked it to the fact that bison's m4sugar.m4 is three years old (it is 
newer than the m4sugar shipped with autoconf 2.59, but older than 2.59c), which 
means it relies on non-POSIX LIFO m4wrap semantics (autoconf 2.60 was the first 
non-beta release that works regardless of LIFO vs. FIFO m4wrap).  ChangeLog-
wise, bison's version looks like it forked from autoconf commit 
http://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=commitdiff;h=cf3b3ede, 
then added the macro m4_prepend (which autoconf should also consider adding).

In addition to Autoconf 2.62's m4sugar already working around the m4wrap issue, 
recent autoconf also has a number of speed optimizations and bug fixes in 
various m4sugar macros, which make autoconf execution noticeably faster than 
what it was with autoconf 2.61 and earlier.  The latest autoconf m4sugar 
provides a speed increase even when using GNU m4 1.4.5, independent from the 
speedups also possible by upgrading to m4 1.6.  However, in improving m4sugar 
for autoconf 2.62, I also made some semantic changes on macros that were 
previously undocumented.  I'm not sure whether blindly replacing 
data/m4sugar/m4sugar.m4 with the latest version from autoconf, modulo the 
addition of m4_prepend and the change of one line regarding the sourcing of 
m4sugar/version.m4, will break anything in bison, since I'm not sure how many 
of the undocumented m4sugar macros were in use by bison.  But something needs 
to be done to ensure that bison will continue to work after m4 1.6 is released, 
and to make it easier to keep m4sugar in sync between bison and autoconf (or 
perhaps even move m4sugar into GNU m4 as the master).  At any rate, I'm 
debating about making bison a show-stopper for the release of m4 1.6 (ie. I'm 
reluctant to release m4 1.6 today, since it would mean the latest released 
versions of bison and m4 would be incompatible).  Autoconf has already proven 
that m4sugar can be made to work across the m4 1.4.x/1.6 transition, so it 
would be nice to have a released version of bison that does likewise, allowing 
a user to upgrade to m4 1.6 without breaking bison (and so that if a user 
happens across a bison/m4 pair that fail to work together, the answer on either 
mailing list can be a simple "known issue; upgrade to the latest stable 
version").

Also, I'm considering making m4 1.6 treat the -d command-line option in a 
position-dependent manner.  That means that bison's current use of 'm4 
m4sugar/m4sugar.m4 - bison.m4 skeleton.m4 -dV" in output_skeleton 
(src/output.c) will fail to turn on the desired tracing, since the -dV occurs 
too late in the command line.  If you want full debugging for anything passed 
into the pipe to m4, the -dV should precede the "-" argument (although it can 
probably occur after the "m4sugar/m4sugar.m4" argument, since tracing the 
loading of m4sugar is not as interesting).  Autoconf was immune to this issue, 
since autom4te uses -d prior to any files.

Meanwhile, bison uses a check for m4 1.3's ability to use frozen files 
(m4/m4.m4, also borrowed from autoconf), even though bison itself does not 
appear to use frozen files.  I don't know how well the current BSD m4 would 
fare with bison's m4 usage patterns, so I suspect that the test is bogus 
(feature-based tests to require GNU m4 over any other version should only be 
necessary if you actually rely on that feature).  On the other hand, I know 
that the BSD folks have already taken pains to implement all GNU extensions 
that were in use by autoconf 2.59 except for frozen files, such that the BSD 
packaging of autoconf intentionally bypasses the m4/m4.m4 check to favor the 
BSD m4 by making a one-line tweak to autom4te to avoid the need for frozen 
files (at the expense of slower operation when using BSD m4 instead of GNU 
m4).  I also know that while updating autoconf's version of m4sugar for 
autoconf 2.62, I took some liberties to use GNU-specific constructs that 
require GNU m4 1.4.5 or newer, but which will fail with the current release of 
BSD m4.  Thus, upgrading to a newer m4sugar may slightly hurt portability to 
BSD systems if the BSD m4 could previously be used to perform the processing 
needed by bison's m4sugar, and where GNU m4 has not been installed.  But you 
can also argue that one of the features that will be introduced by GNU m4 1.6 
(linear instead of quadratic execution speed on recursive macro expansion) 
means that introducing a hard dependency on GNU m4 is in the user's best 
interest, since BSD m4 already suffers from slower performance, and the 
performance gap between GNU and BSD only widens with the newer GNU m4 release.

I am willing to file copyright papers and help in the effort of updating 
bison's usage of m4sugar, if you would like, since I maintain GNU m4 and made 
the bulk of the autoconf m4sugar improvements over the past 3 years.

-- 
Eric Blake