Parallel tests execution [0/4]

[Ralf Wildenhues]

Hello everyone,

These four patches implement parallel execution of TESTS in Automake,
adapted from the check.mk file Akim Demaille posted earlier.  I would
be delighted about review, comments, bug reports, any feedback.  There
are a couple of questions at the end.

Design considerations:

1) should be similar to current TESTS interface, and not enabled without
   developer consent (because of inevitable semantic differences)
2) should allow running of a subset of tests easily, preferably as
     make check TESTS="..."
3) should work with different test naming styles:
     - *.test (with log files named s/\.test$/.log/)
     - no particular suffix (log files will be s/$/.log/)
     - should work with executables (e.g., TESTS = $(check_PROGRAMS))
     - arbitrary other suffixes
4) should work with @substituted@ values in TESTS,
   (here a limitation upon nonempty suffixes is ok)
5) parallel test execution,
6) lazy test completion (do not rerun already-run tests),
7) portable make (of course!), portable shell, etc.,
8) should allow for hard errors that stop test suite execution,
9) should allow for dependency relations (ordering) between tests,
10) test names may have directory components,
------
11) allow for extensibility like multiple test suites, each with their
    own set of logs.
12) allow for additional output on stdout/stderr?
    example: test was skipped because of $reason.
13) allow for post-test action in the rule (like: clean foo.dir in
    Automake's test suite)?
14) let subdir tests depend on dirstamp instead of each one calling
    MKDIR_P?

The last items have not been addressed.


Implementation notes:

- (1) is addressed by using a new Automake option 'parallel-tests', and
  checking that both old and new driver pass all common tests (each one
  is run twice).
- (2) precludes that we reuse the _SOURCES interface, unfortunately.
- (2) and (3) preclude that the user can specify something like
    TEST_LOGS = $(TESTS:.test=.log)
  Instead, we let her specify
    TEST_SUFFIXES = .test
  We could also have tried to gather all kinds of suffixes from $(TESTS)
  but that may lead to more ambiguities in suffix rules that desirable
- if there is a nonempty test suffix, we can use suffix rules and avoid
  a per-target rule for each test.  That means @substituted@ values only
  work if the test suffix is nonempty (or we would have to use the hack
  from check.mk that restarts make with overwritten variables).
- (5), (6), (8) are provided already by the check.mk code, except that
  (8) didn't work.


Changes over the check.mk code:

- Replace $(cmd)  by `cmd` (for Solaris)
- stick to portable awk (for Solaris)
- fix awk script to not drop output
- no white space at beginning or end of summary output
- sed: use literal TAB instead of \t in regex
- no leading underscore in make macro names
- use $(AM_MAKEFLAGS) whereever $(MAKE) is used
- also clean *.log-t intermediate files
- clean rules merged into automake proper (%clean_files)
- transform pattern rules into suffix or per-target rules rules, for
  non-GNU make (and avoid $< in the non-suffix rules)
- avoid Solaris make VPATH rewriting bugs
- output `PASS: foo.test' not `PASS: foo.log'
  but respect subdirs
- some optimization (avoid lots of `sed 1q')
- make hard errors work; use exit status 99 as indicator (values >128
  are not portable, and should not be close to 128); variable used it
  DISABLE_HARD_ERRORS, not ENABLE_HARD_ERRORS, so that the default can
  be unset (which is better for portably transporting a user override
  across recursive makes).
- Fix summary not to count SKIPs as PASSes
- Fix singular/plural in summary


Questions, and issues I'm not sure about yet:

- improvements on the naming scheme would be good; some of the new
  variables seem quite ad-hoc

- shouldn't test-suite.log also be created if all tests passed?

- should skipped tests show up in test-suite.log?

- is everybody ok with the following authorship for patch 1/4?
  2008-10-XX  Akim Demaille <address@hidden>
              Jim Meyering <address@hidden>
              Benoit Sigoure <address@hidden>
              Ralf Wildenhues <address@hidden>

  (as git allows only one author, I will put the first name in --author)

- how should I best acknowledge The Vaucanson Group?  Something like
this in lib/am/check.am ok?

## This code is adapted from check.mk which came from:
##
## Vaucanson, a generic library for finite state machines.
## Copyright (C) 2006, 2007 The Vaucanson Group.

The only reason I haven't put this in yet is that it would require a
copyright disclaimer from Vaucanson.  What do you think?


Cheers, and thanks for reading this far,
Ralf

1/4    Parallel test execution: new option `parallel-tests'.
2/4    parallel-tests: Ensure backward-compatible semantics.
3/4    New tests for `parallel-tests'.
4/4    Documentation for the parallel-tests driver.