Re: Autotest: loops of tests

[Ralf Wildenhues]

Hi Noah,

* Noah Misch wrote on Tue, Aug 16, 2005 at 12:10:32AM CEST:
> On Mon, Aug 15, 2005 at 06:26:29PM +0200, Ralf Wildenhues wrote:
> > Currently, AT_CHECK(cmd, ignore, , , fail_commands)
> > is somewhat superfluous, as fail_commands will never get executed
> > because we ignore the return value.  The patch below changes that.
> > I regard this change in semantics as ok because of the nonsensical
> > behavior this combination had before, IMVHO.
> 
> AFAICS, fail_commands would run if cmd has a non-empty stdout or stderr.

You mean, if $3 or $4 of AT_CHECK contain the expected output, right?

Then yes, fail_commands would run.  But I can't predict the output.
Libtool link output may vary greatly depending on the system used,
that's exactly its function.  It's also virtually impossible to match
this against a few known cases.

> > ### snip loop.at ###
> > dnl autom4te -l autotest loop.at > loop; chmod +x loop; ./loop
> > m4_define([AT_PACKAGE_STRING],[loop])
> > m4_define([AT_PACKAGE_BUGREPORT],[devnull])
> > AT_INIT
> > AT_SETUP([loop])
> > AT_CAPTURE_FILE([failures])
> > for i in '' one '' two '' three; do
> >   dnl Stylistic issue: we overwrite `failures' here to avoid quadratic
> >   dnl growth of `testsuite.log' in the number of failures.
> >   dnl This is because many tests might fail for the same reason.
> >   AT_CHECK([test -z "$i"], ignore,ignore,ignore, [echo "$i" > failures])
> 
> Does replacing with this obviate the need for your patch?
> 
> AT_CHECK([test -z "$i" || echo "$i" >failures], ignore, ignore, ignore)

That is not allowed:

|  -- Macro: AT_CHECK (COMMANDS, [STATUS = ``0''], [STDOUT = ``''],
|           [STDERR = ``''], [RUN-IF-FAIL], [RUN-IF-PASS])
*snip*
| 
|      The COMMANDS _must not_ redirect the standard output, nor the
|      standard error.

Otherwise, I'd guess this would be ok.  But I also like the feature of
allowing files to be captured the way I proposed.  Obviates the need to
write even more of my own test suite inside autotest.  :)

> > done
> > 
> > dnl next command generates the real failure:
> > AT_CHECK([test -s failures],1,ignore,ignore)
> 
> This does look like a sound technique for testing many variations of a 
> command.

Exactly.  So, does it have a chance of ending up in Autotest?

Cheers,
Ralf