Re: autotest annoyances

[Ralf Wildenhues]

Hi Eric,

* Eric Blake wrote on Sat, Apr 08, 2006 at 03:16:33AM CEST:
> While testing my previous patch for const/volatile vs. -Werror, I encountered
> two autotest anomalies.  I first enhanced the testsuite to detect both
> problems, as well as my earlier patch this week for when removing
> the test group directory fails.

It's not completely obvious to me that your tests also detect the
earlier failure.  Is that because they happen to use `./run' at all?
If so, then that is ok.

> The first bug was that when using the -d option to testsuite (which is done by
> all the testsuite.dir/###/run scripts), the run script was only regenerated
> if the test failed (it was missing after a success or skip).

Hmm.  Both `./testsuite --help' and the documentation state explicitly
that `-d' prevents creation of the `run' script in the first place.
I see that this may not be the best idea to keep that, since I don't
know why using `./run' once should not be idempotent, i.e., result in
the same state after rerunning the test.  But anyway the documentation
needs to be fixed; but first we should come up with a sane set of
semantics.

> The second bug is that environment variables passed as options to ./testsuite
> are copied directly into the run scripts, but without regards for shell
> metacharacters.  That means that an environment variable that contains
> spaces, such as "./run CFLAGS='-std=c89 -Werror'", will regenerate a broken
> run script that tries to invoke 'testsuite CFLAGS=-std=c89 -Werror'.  However,
> fixing this quickly and safely is beyond my abilities, with a pending alpha
> release, so the test is just xfailed for now.

This could be fixed similarly to how lib/autoconf/general.m4 quotes
the contents of ac_configure_args selectively for config.status.
A bit tricky, but it's possible.  (It's not safe against all kinds
of uses of backslashes; that would require searching for an echo
replacement like Libtool does.)

Cheers,
Ralf