Re: Parallel tests execution [0/4]

[Jim Meyering]

Ralf Wildenhues <address@hidden> wrote:
> Hello everyone,
>
> These four patches implement parallel execution of TESTS in Automake,
> adapted from the check.mk file Akim Demaille posted earlier.  I would
> be delighted about review, comments, bug reports, any feedback.  There
> are a couple of questions at the end.

Great!
Thanks for all the work, Ralf.

> Design considerations:
>
> 1) should be similar to current TESTS interface, and not enabled without
>    developer consent (because of inevitable semantic differences)
> 2) should allow running of a subset of tests easily, preferably as
>      make check TESTS="..."
> 3) should work with different test naming styles:
>      - *.test (with log files named s/\.test$/.log/)
>      - no particular suffix (log files will be s/$/.log/)
>      - should work with executables (e.g., TESTS = $(check_PROGRAMS))
>      - arbitrary other suffixes
> 4) should work with @substituted@ values in TESTS,
>    (here a limitation upon nonempty suffixes is ok)
> 5) parallel test execution,
> 6) lazy test completion (do not rerun already-run tests),
> 7) portable make (of course!), portable shell, etc.,
> 8) should allow for hard errors that stop test suite execution,
> 9) should allow for dependency relations (ordering) between tests,
> 10) test names may have directory components,
> ------
> 11) allow for extensibility like multiple test suites, each with their
>     own set of logs.
> 12) allow for additional output on stdout/stderr?
>     example: test was skipped because of $reason.

This would be nice.
It'd would be useful also to mark as "surprising" or
"highly undesirable" the results of certain tests.
Better than a simple "yes" or "no".

...
> Questions, and issues I'm not sure about yet:
>
> - improvements on the naming scheme would be good; some of the new
>   variables seem quite ad-hoc
>
> - shouldn't test-suite.log also be created if all tests passed?

Sounds reasonable.

> - should skipped tests show up in test-suite.log?

I'd find that useful.

I'll give it a spin in coreutils soon.