Re: [GNU Autoconf 2.71] testsuite: 10 11 12 14 37 92 236 238 239 240 293 299 397 430 431 432 433 434 435 436 437 438 439 440 509 514 failed

[Zack Weinberg]

On Sat, Jul 2, 2022, at 5:27 PM, Tom Lane wrote:
> I thought I'd respond to the question raised at [1] about which
> versions of m4 have this problem.
...
> I could build most versions on NetBSD 9.2/amd64,
...
> m4-1.4.7 shows largely the same set of failures as with 1.4.6
...
> With m4-1.4.8, there are just two failing tests

This is super helpful.  I had been assuming it was a bad Apple patch ultimately 
at fault, but I can reproduce more-or-less the same set of failures with m4 
1.4.[67] on my Linux workstation, in enough detail to investigate.

It turns out that there are three underlying issues.  One is a buggy regexp in 
Autoconf's testsuite that does not do what the comments say it does, causing 
any test that inspects m4 error messages to fail spuriously due to harmless 
variation in the wording between m4 versions.  I checked in commit 5d8a59e3 to 
fix this.

The other two issues are genuine problems with m4 1.4.6 and 1.4.7:

1) This command

    printf '%s\n' \
        'changequote(<,>)define(<T>,<>)dnl' \
        'define(<F>,<T(<traced>)>)dnl' \
        'm4wrap(<F>)dnl' \
    | m4 --debug=aflq --trace=T

prints

    m4trace:stdin:3: -1- T(<traced>)

with m4 >= 1.4.8, but

    m4trace: -1- T(<traced>)

with 1.4.6 and 1.4.7.  The 'autom4te' wrapper script expects all 'm4trace:' 
lines to start with a file name and a line number, so it fails to parse the 
output.  This is the root cause of roughly half the testsuite failures.

2) This command

    printf '%s\n' \
        'changequote(<,>)define(<T>,<__line__>)dnl' \
        'T(<' \
        'abc' \
        '>)' \
    | m4

prints '2' with m4 >= 1.4.8 but '4' with 1.4.6 and 1.4.7.  That breaks the 
other half of the failing tests, which are all looking at autoconf diagnostic 
output and care about what line number is associated with a problem.

If it was just (2) I might have considered fixing it in the testsuite, but (1) 
is serious enough that I do think desupporting 1.4.6 and 1.4.7 is justified.

The NEWS file for m4 1.4.8 mentions both of these as deliberate changes:

# * Warnings and errors issued during macro expansion are now consistently
#   reported at the line where the macro name was detected, rather than
#   where the close parenthesis resides.  Text wrapped by `m4wrap' now
#   remembers the location that was in effect when m4wrap was invoked,
#   rather than changing to line 0 and the empty string for a file.

I have checked in commit 5d8a59e3 to make configure reject 1.4.6 and 1.4.7, and 
verified that, on my computer, I get 100% test successes with 1.4.8 and 1.4.19 
(again, this is Linux) after both patches are applied.  I also verified that 
configure does fail when 1.4.6 is the only version of m4 available.

The content of the patches is attached to this message.

> I soon found that GNU m4 is an incredibly fragile, system-dependent
> mess.

Yeah, with all the good will in the world, I don't think quadratic strstr() or 
lack of support for fseeko() are things that an application should be trying to 
work around.  Get the C library fixed instead!

...
> 1.4.18 does build, and we're back to
>
>    Subject: [GNU Autoconf 2.71] testsuite: 260 269 failed

I _would_ like to see the testsuite.log files from this run, specifically.  NB 
The test numbers change every time someone adds a new test, so it's not super 
helpful to know that "260 and 269" are failing consistently, with no further 
information.

I'll follow up on the AC_HEADER_STDBOOL failures on macOS in a separate message 
(and possibly not for a few days).

> My recommendation would actually be to take out the check in m4.m4
> that purports to reject buggy strstr behavior --- if that's looking
> for something real, you couldn't tell it from the test suite.

It's a very real bug: the `index` builtin fails to find substrings that are 
actually present.  `index` (renamed `m4_index` for namespace cleanliness) is 
used in dozens of places in autoconf's code, so if the testsuite passes with 
the bug present, it's only by dumb luck.

zw

0002-Require-GNU-M4-1.4.8-or-later.patch
Description: Text Data

0001-testsuite-Handle-balanced-ASCII-quotes-correctly-in-.patch
Description: Text Data