Re: [PATCH] parallel-tests: new recognized test result 'ERROR'

[Ralf Wildenhues]

* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 04:58:21PM CEST:
> I've applied the attached patch to the 'GSoC/experimental/test-results-work'
> branch  Note that this is not an FYI, since that branch is temporary, and can
> thus be amended and modified.  Reviews are welcome!

Just a couple of quick thoughts:
- is ERROR the same term other testing frameworks use for similar
  semantics?  Let's strive to be consistent.
- will users confuse FAIL and ERROR and try to use the latter where the
  former is appropriate (or vice versa)?

Let's try to avoid inventing different terms where and when we can.

Changing semantics requires a NEWS entry.  It's actually an
incompatibility, but a very minor one, so it's not a big problem.

Thanks,
Ralf

> Subject: [PATCH] parallel-tests: new recognized test result 'ERROR'
> 
> * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
> `ERROR'.  Use it when encountering unreadable test logs (previously
> a simple `FAIL' was used in this situations).
> * lib/test-driver: Set the global test result to `ERROR' when the
> test exit status is 99.  When doing colorized output, color `ERROR'
> results in magenta.
> * doc/automake.texi (Log files generation and test results
> recording): Update by also listing `ERROR' among the list of valid
> `:test-results:' arguments.
> * tests/trivial-test-driver: Update.
> * tests/parallel-tests.test: Likewise.
> * tests/parallel-tests-harderror.test: Likewise.
> * tests/parallel-tests-no-spurious-summary.test: Likewise.
> * tests/test-driver-global-log.test: Likewise.
> * tests/test-driver-recheck.test: Likewise.
> * tests/test-driver-custom-multitest-recheck.test: Likewise.
> * tests/test-driver-custom-multitest-recheck2.test: Likewise.
> * tests/test-driver-custom-multitest.test: Likewise.
> * tests/test-driver-custom-no-html.test: Likewise.
> * tests/test-driver-end-test-results.test: Likewise.
> * tests/color.test: Likewise, and make stricter.
> * tests/color2.test: Likewise, and improve syncing with color.test.
> * tests/parallel-tests-exit-statuses.test: New test.
> * tests/parallel-tests-console-output.test: New test.
> * tests/Makefile.am (TESTS): Update.

> --- a/doc/automake.texi
> +++ b/doc/automake.texi
> @@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
>  
>  @c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
>  The only recognized test results are currently @code{PASS}, @code{XFAIL},
> address@hidden, @code{FAIL} and @code{XPASS}.  These results, when declared
> -with @code{:test-result:}, can be optionally followed by text holding
> -the name and/or a brief description of the corresponding test; the
> address@hidden harness will ignore such extra text when
> address@hidden, @code{FAIL}, @code{XPASS} and @code{ERROR}.  These results,
> +when declared with @code{:test-result:}, can be optionally followed by
> +text holding the name and/or a brief description of the corresponding
> +test; the @option{parallel-tests} harness will ignore such extra text when

ERROR semantics are not actually described.

>  generating @file{test-suite.log} and preparing the testsuite summary.
>  Also, @code{:test-result:} can be used with a special ``pseudo-result''
>  @code{END}, that will instruct the testsuite harness to stop scanning

> --- a/lib/am/check.am
> +++ b/lib/am/check.am
> @@ -66,7 +66,7 @@ include inst-vars.am
>  ## appended.
>  ##
>  ## In addition to the magic "exit 77 means SKIP" feature (which was
> -## imported from automake), there is a magic "exit 99 means FAIL" feature
> +## imported from automake), there is a magic "exit 99 means ERROR" feature
>  ## which is useful if you need to issue a hard error no matter whether the
>  ## test is XFAIL or not.  You can disable this feature by setting the
>  ## variable DISABLE_HARD_ERRORS to a nonempty value.
> @@ -153,7 +153,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
>  ## Readable test logs.
>       list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
>  ## Each unreadable test log counts as a failed test.
> -     results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
> +     results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
>  ## Now we're going to extract the outcome of all the testcases from the
>  ## test logs.
>       results2=''; \
> @@ -187,6 +187,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
>       skip=`echo "$$results" | grep -c '^SKIP'`;                      \
>       xfail=`echo "$$results" | grep -c '^XFAIL'`;                    \
>       xpass=`echo "$$results" | grep -c '^XPASS'`;                    \
> +     error=`echo "$$results" | grep -c '^ERROR'`;                    \
> +## FIXME: for the moment, we count errors as failures, otherwise the code
> +## that display the testsuite summary will become too much complicated.

grammar nits: s/display/&s/; s/much //

> +     fail=`expr $$fail + $$error`;                                   \
>       failures=`expr $$fail + $$xpass`;                               \
>       all=`expr $$all - $$skip`;                                      \
>       if test "$$all" -eq 1; then tests=test; All=;                   \

> --- /dev/null
> +++ b/tests/parallel-tests-console-output.test

> +# parallel-tests: some checks on console output about testsuite
> +# advancement.

What do you mean by "advancement"?  Maybe "progress" or "output order!?

> +parallel_tests=yes
> +. ./defs || Exit 1
> +
> +cat >> configure.in << 'END'
> +AC_OUTPUT
> +END
> +
> +cat > Makefile.am << 'END'
> +XFAIL_TESTS = sub/xpass.test xfail.test error.test
> +TESTS = $(XFAIL_TESTS) fail.test pass.test a/b/skip.test sub/error2.test
> +pass.log: fail.log
> +error.log: pass.log
> +sub/xpass.log: error.log
> +sub/error2.log: xfail.log
> +a/b/skip.log: sub/error2.log
> +END
> +
> +cat > exp <<'END'
> +FAIL: fail.test
> +PASS: pass.test
> +ERROR: error.test
> +XPASS: sub/xpass.test
> +XFAIL: xfail.test
> +ERROR: sub/error2.test
> +SKIP: a/b/skip.test
> +END
> +
> +mkdir sub a a/b
> +
> +cat > pass.test << 'END'
> +#!/bin/sh
> +exit 0
> +END
> +cp pass.test sub/xpass.test
> +
> +cat > fail.test << 'END'
> +#!/bin/sh
> +exit 1
> +END
> +
> +cat > xfail.test << 'END'
> +#!/bin/sh
> +# The sleep should ensure expected execution order of tests
> +# even when make is run in parallel mode.
> +# FIXME: quotes below required by maintainer-check
> +sleep '10'

This looks like it would take long and/or be racy?

> +exit 1
> +END
> +
> +cat > error.test << 'END'
> +#!/bin/sh
> +exit 99
> +END
> +cp error.test sub/error2.test
> +
> +cat > a/b/skip.test << 'END'
> +#!/bin/sh
> +exit 77
> +END
> +
> +chmod a+x pass.test fail.test xfail.test sub/xpass.test \
> +          a/b/skip.test error.test sub/error2.test
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE -a
> +
> +for vpath in : false; do
> +  if $vpath; then
> +    mkdir build
> +    cd build
> +    srcdir=..
> +  else
> +    srcdir=.
> +  fi
> +  $srcdir/configure
> +  $MAKE check >stdout && { cat stdout; Exit 1; }
> +  cat stdout
> +  LC_ALL=C grep '^[A-Z][A-Z]*:' stdout > got
> +  cat got
> +  diff $srcdir/exp got
> +  cd $srcdir
> +done

> --- /dev/null
> +++ b/tests/parallel-tests-exit-statuses.test

> +# Check parallel-tests features: normal and special exit statuses
> +# in the test scripts.
> +
> +parallel_tests=yes
> +. ./defs || Exit 1
> +
> +cat >> configure.in << 'END'
> +AC_OUTPUT
> +END
> +
> +# $failure_statuses should be defined to the list of all integers between
> +# 1 and 255 (inclusive), excluded 77 and 99.

Actually, I wouldn't go over 125 for portability reasons.

> +# Let's use `seq' if available, it's faster than the loop.
> +failure_statuses=`seq 1 255 \

`(seq 255) 2>/dev/null \

so that the error from nonpresent seq does not escape even with
ksh/dash.

> +  || { i=1; while test $i -le 255; do echo $i; i=\`expr $i + 1\`; done; }`
> +failure_statuses=`
> +  for i in $failure_statuses; do
> +    test $i -eq 77 || test $i -eq 99 || echo $i
> +  done | tr "$nl" ' '`
> +# For debugging.
> +echo "failure_statuses: $failure_statuses"
> +# Sanity check.
> +test `for st in $failure_statuses; do echo $st; done | wc -l` -eq 253 \
> +  || fatal_ "initializing list of exit statuses for simple failures"
> +
> +cat > Makefile.am <<END
> +LOG_COMPILER = ./do-exit
> +fail_tests = $failure_statuses
> +TESTS = 0 77 99 $failure_statuses
> +\$(TESTS):
> +END
> +
> +cat > do-exit <<'END'
> +#!/bin/sh
> +echo "$0: $1"
> +case $1 in
> +  [0-9]|[0-9][0-9]|[0-9][0-9][0-9]) st=$1;;
> +  */[0-9]|*/[0-9][0-9]|*/[0-9][0-9][0-9]) st=`echo x"$1" | sed 's|.*/||'`;;
> +  *) st=99;;
> +esac
> +exit $st
> +END
> +chmod a+x do-exit
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE -a
> +
> +{
> +  echo PASS: 0
> +  echo SKIP: 77
> +  echo ERROR: 99
> +  for st in $failure_statuses; do
> +    echo "FAIL: $st"
> +  done
> +} | LC_ALL=C sort > exp-fail
> +
> +sed 's/^FAIL:/XFAIL:/' exp-fail | LC_ALL=C sort > exp-xfail-1
> +sed '/^ERROR:/d' exp-xfail-1 > exp-xfail-2
> +
> +sort exp-fail
> +sort exp-xfail-1
> +sort exp-xfail-2
> +
> +./configure
> +
> +st=1
> +$MAKE check >stdout && st=0
> +cat stdout
> +cat test-suite.log
> +test $st -gt 0 || Exit 1
> +LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-fail
> +diff exp-fail got-fail
> +
> +st=1
> +XFAIL_TESTS="$failure_statuses 99" $MAKE -e check >stdout && st=0
> +cat stdout
> +cat test-suite.log
> +test $st -gt 0 || Exit 1
> +LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-1
> +diff exp-xfail-1 got-xfail-1
> +
> +st=0
> +XFAIL_TESTS="$failure_statuses" TESTS="0 77 $failure_statuses" \
> +  $MAKE -e check >stdout || st=$?
> +cat stdout
> +cat test-suite.log
> +test $st -eq 0 || Exit 1
> +LC_ALL=C grep '^[A-Z][A-Z]*:' stdout | LC_ALL=C sort > got-xfail-2
> +diff exp-xfail-2 got-xfail-2
> +
> +:
> diff --git a/tests/parallel-tests-harderror.test 
> b/tests/parallel-tests-harderror.test
> index 8680e29..f30fd52 100755
> --- a/tests/parallel-tests-harderror.test
> +++ b/tests/parallel-tests-harderror.test
> @@ -69,13 +69,13 @@ DISABLE_HARD_ERRORS=x $MAKE check
>  
>  $MAKE check DISABLE_HARD_ERRORS='' && Exit 1
>  cat test-suite.log
> -grep '^FAIL: foo\.test .*exit.*99' test-suite.log
> +grep '^ERROR: foo\.test .*exit.*99' test-suite.log
>  
>  cd sub
>  # The `-e' is wanted here.
>  DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
>  cat test-suite.log
> -grep '^FAIL: bar\.test .*exit.*99' test-suite.log
> +grep '^ERROR: bar\.test .*exit.*99' test-suite.log
>  cd ..
>  
>  # Check the distributions.
> @@ -90,7 +90,7 @@ $AUTOMAKE Makefile
>  ./config.status Makefile
>  VERBOSE=yes $MAKE check && Exit 1
>  grep '^FAIL' test-suite.log && Exit 1
> -grep '^FAIL: bar\.test .*exit.*99' sub/test-suite.log
> +grep '^ERROR: bar\.test .*exit.*99' sub/test-suite.log
>  
>  echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
>  VERBOSE=yes $MAKE check
> diff --git a/tests/parallel-tests-no-spurious-summary.test 
> b/tests/parallel-tests-no-spurious-summary.test
> index 43c54a5..8b9547a 100755
> --- a/tests/parallel-tests-no-spurious-summary.test
> +++ b/tests/parallel-tests-no-spurious-summary.test
> @@ -32,10 +32,12 @@ cat > foo.test <<'END'
>  #! /bin/sh
>  echo :test-result:XFAIL
>  echo :test-result: SKIP
> +echo :test-result:ERROR
>  exit 0
>  END
>  cat > bar.test <<'END'
>  #! /bin/sh
> +echo :test-result: ERROR
>  echo :test-result:FAIL
>  echo :test-result: XPASS
>  exit 0
> diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test
> index f046c3f..95f78e7 100755
> --- a/tests/parallel-tests.test
> +++ b/tests/parallel-tests.test
> @@ -68,10 +68,13 @@ $AUTOMAKE -a
>  
>  $MAKE check >stdout && { cat stdout; Exit 1; }
>  cat stdout
> -# There should be two errors: bar.test is a hard error.
> -test `grep -c '^FAIL' stdout` -eq 2
> +# There should be one failure and one hard error.
> +test `grep -c '^FAIL:' stdout` -eq 1
> +test `grep -c '^ERROR:' stdout` -eq 1
>  test -f mylog.log
> -test `grep -c '^FAIL' mylog.log` -eq 2
> +cat mylog.log
> +test `grep -c '^FAIL:' mylog.log` -eq 1
> +test `grep -c '^ERROR:' mylog.log` -eq 1
>  test -f baz.log
>  test -f bar.log
>  test -f foo.log
> diff --git a/tests/test-driver-custom-multitest-recheck.test 
> b/tests/test-driver-custom-multitest-recheck.test
> index e0f010f..c178fa0 100755
> --- a/tests/test-driver-custom-multitest-recheck.test
> +++ b/tests/test-driver-custom-multitest-recheck.test
> @@ -49,7 +49,7 @@ echo PASS:
>  if test -f b.ok; then
>    echo PASS:
>  else
> -  echo FAIL:
> +  echo ERROR:
>  fi
>  : > b.run
>  END
> @@ -97,14 +97,15 @@ do_recheck ()
>  
>  do_count ()
>  {
> -  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
> +  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
>    eval "$@"
> -  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
> +  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
>    test `grep -c '^PASS:' stdout` -eq $pass
>    test `grep -c '^FAIL:' stdout` -eq $fail
>    test `grep -c '^XPASS:' stdout` -eq $xpass
>    test `grep -c '^XFAIL:' stdout` -eq $xfail
>    test `grep -c '^SKIP:' stdout` -eq $skip
> +  test `grep -c '^ERROR:' stdout` -eq $error
>  }
>  
>  for vpath in : false; do
> @@ -129,7 +130,7 @@ for vpath in : false; do
>    test ! -r c.log
>    test ! -r d.run
>    test ! -r d.log
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>  
>    : Run the tests for the first time.
>    $MAKE check >stdout && { cat stdout; Exit 1; }
> @@ -140,7 +141,7 @@ for vpath in : false; do
>    test -f b.run
>    test -f c.run
>    test -f d.run
> -  do_count pass=3 fail=3 xpass=1 xfail=1 skip=1
> +  do_count pass=3 fail=2 xpass=1 xfail=1 skip=1 error=1
>  
>    : Let us make b.test pass.
>    echo OK > b.ok
> @@ -151,7 +152,7 @@ for vpath in : false; do
>    test -f b.run
>    test -f c.run
>    test -f d.run
> -  do_count pass=2 fail=2 xpass=1 xfail=1 skip=1
> +  do_count pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
>  
>    : Let us make the first part of c.test pass.
>    echo OK > c.pass
> @@ -160,7 +161,7 @@ for vpath in : false; do
>    test ! -r b.run
>    test -f c.run
>    test -f d.run
> -  do_count pass=1 fail=1 xpass=1 xfail=1 skip=1
> +  do_count pass=1 fail=1 xpass=1 xfail=1 skip=1 error=0
>  
>    : Let us make also the second part of c.test pass.
>    echo KO > c.xfail
> @@ -169,7 +170,7 @@ for vpath in : false; do
>    test ! -r b.run
>    test -f c.run
>    test -f d.run
> -  do_count pass=1 fail=1 xpass=0 xfail=2 skip=1
> +  do_count pass=1 fail=1 xpass=0 xfail=2 skip=1 error=0
>  
>    : Nothing changed, so only d.test should be run.
>    for i in 1 2; do
> @@ -178,7 +179,7 @@ for vpath in : false; do
>      test ! -r b.run
>      test ! -r c.run
>      test -f d.run
> -    do_count pass=0 fail=1 xpass=0 xfail=0 skip=1
> +    do_count pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
>    done
>  
>    : Let us make d.test run more testcases, and experience _more_ failures.
> @@ -190,14 +191,16 @@ for vpath in : false; do
>      echo XPASS: xp
>      echo FAIL: f 3
>      echo FAIL: f 4
> +    echo ERROR: e 1
>      echo PASS: p 2
> +    echo ERROR: e 2
>  END
>    do_recheck --fail
>    test ! -r a.run
>    test ! -r b.run
>    test ! -r c.run
>    test -f d.run
> -  do_count pass=2 fail=4 xpass=1 xfail=0 skip=2
> +  do_count pass=2 fail=4 xpass=1 xfail=0 skip=2 error=2
>  
>    : Let us finally make d.test pass.
>    echo : > d.extra
> @@ -206,7 +209,7 @@ END
>    test ! -r b.run
>    test ! -r c.run
>    test -f d.run
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=1
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=1 error=0
>  
>    : All tests have been successful or skipped, nothing should be re-run.
>    do_recheck --pass
> @@ -214,7 +217,7 @@ END
>    test ! -r b.run
>    test ! -r c.run
>    test ! -r d.run
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>  
>    cd $srcdir
>  
> diff --git a/tests/test-driver-custom-multitest-recheck2.test 
> b/tests/test-driver-custom-multitest-recheck2.test
> index a18bdd8..d2e979e 100755
> --- a/tests/test-driver-custom-multitest-recheck2.test
> +++ b/tests/test-driver-custom-multitest-recheck2.test
> @@ -56,7 +56,13 @@ END
>  
>  cat > c.test << 'END'
>  #! /bin/sh
> -echo XPASS: xp
> +if test -f c.err; then
> +  echo ERROR: xxx
> +elif test -f c.ok; then
> +  echo PASS: ok
> +else
> +  echo XPASS: xp
> +fi
>  : > c.run
>  END
>  
> @@ -68,14 +74,15 @@ $AUTOMAKE
>  
>  do_count ()
>  {
> -  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR
> +  pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR
>    eval "$@"
> -  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout || : # For debugging.
> +  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout || : # For debugging.
>    test `grep -c '^PASS:' stdout` -eq $pass
>    test `grep -c '^FAIL:' stdout` -eq $fail
>    test `grep -c '^XPASS:' stdout` -eq $xpass
>    test `grep -c '^XFAIL:' stdout` -eq $xfail
>    test `grep -c '^SKIP:' stdout` -eq $skip
> +  test `grep -c '^ERROR:' stdout` -eq $error
>  }
>  
>  for vpath in : false; do
> @@ -96,7 +103,7 @@ for vpath in : false; do
>    test -f a.run
>    test -f b.run
>    test -f c.run
> -  do_count pass=2 fail=1 xpass=1 xfail=0 skip=1
> +  do_count pass=2 fail=1 xpass=1 xfail=0 skip=1 error=0
>  
>    rm -f *.run
>  
> @@ -104,7 +111,7 @@ for vpath in : false; do
>    for var in TESTS TEST_LOGS; do
>      env "$var=" $MAKE -e recheck >stdout || { cat stdout; Exit 1; }
>      cat stdout
> -    do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +    do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>      test ! -r a.run
>      test ! -r b.run
>      test ! -r c.run
> @@ -115,7 +122,7 @@ for vpath in : false; do
>    env TESTS=a.test $MAKE -e recheck >stdout \
>      || { cat stdout; Exit 1; }
>    cat stdout
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>    test ! -r a.run
>    test ! -r b.run
>    test ! -r c.run
> @@ -128,7 +135,7 @@ for vpath in : false; do
>    test ! -r a.run
>    test -f b.run
>    test ! -r c.run
> -  do_count pass=0 fail=0 xpass=0 xfail=1 skip=1
> +  do_count pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
>  
>    rm -f *.run
>  
> @@ -136,35 +143,54 @@ for vpath in : false; do
>    TEST_LOGS=b.log $MAKE -e recheck >stdout \
>      || { cat stdout; Exit 1; }
>    cat stdout
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>    test ! -r a.run
>    test ! -r b.run
>    test ! -r c.run
>    TESTS='a.test b.test' $MAKE -e recheck >stdout \
>      || { cat stdout; Exit 1; }
>    cat stdout
> -  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
>    test ! -r a.run
>    test ! -r b.run
>    test ! -r c.run
>  
> -  # An XPASS should count a failure.
> +  : No need to re-run a.test anymore, but c.test should be rerun,
> +  : as it contained an XPASS.  And this time, make it fail with
> +  : an hard error.
> +  : > c.err
>    env TEST_LOGS='a.log c.log' $MAKE -e recheck >stdout \
>      && { cat stdout; Exit 1; }
>    cat stdout
> -  do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1
>    test ! -r a.run
>    test ! -r b.run
>    test -f c.run
> -  rm -f *.run
> -  env TESTS='c.test b.test' $MAKE -e recheck >stdout \
> -    && { cat stdout; Exit 1; }
> +
> +  rm -f *.run *.err
> +
> +  : c.test contained and hard error the last time, so it should be re-run.
> +  : This time, make it pass
> +  : > c.ok
> +  env TESTS='c.test a.test' $MAKE -e recheck >stdout \
> +    || { cat stdout; Exit 1; }
>    cat stdout
> -  do_count pass=0 fail=0 xpass=1 xfail=0 skip=0
> +  do_count pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
>    test ! -r a.run
>    test ! -r b.run
>    test -f c.run
>  
> +  rm -f *.run *.err *.ok
> +
> +  : Nothing should be rerun anymore, as all tests have been evenatually

eventually

> +  : succesfull.

successful

> +  $MAKE recheck >stdout || { cat stdout; Exit 1; }
> +  cat stdout
> +  do_count pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
> +  test ! -r a.run
> +  test ! -r b.run
> +  test ! -r c.run
> +
>    cd $srcdir
>  
>  done
> diff --git a/tests/test-driver-custom-multitest.test 
> b/tests/test-driver-custom-multitest.test
> index a1c1010..2366b5b 100755
> --- a/tests/test-driver-custom-multitest.test
> +++ b/tests/test-driver-custom-multitest.test
> @@ -40,7 +40,7 @@ TESTS = \
>    pass-fail.t \
>    pass4-skip.t \
>    pass3-skip2-xfail.t \
> -  pass-xpass-fail-xfail-skip.t
> +  pass-xpass-fail-xfail-skip-error.t
>  END
>  
>  expected_pass=10
> @@ -48,6 +48,7 @@ expected_fail=5
>  expected_skip=4
>  expected_xfail=2
>  expected_xpass=1
> +expected_error=1
>  
>  cat > pass.t << 'END'
>  echo %% pass %%
> @@ -99,13 +100,14 @@ mv || echo "XFAIL: mv cannot read users' mind yet"
>  exit 127
>  END
>  
> -cat > pass-xpass-fail-xfail-skip.t << 'END'
> +cat > pass-xpass-fail-xfail-skip-error.t << 'END'
>  echo PASS:
>  echo FAIL:
>  echo XFAIL:
>  echo XPASS:
>  echo SKIP:
> -echo %% pass-xpass-fail-xfail-skip %%
> +echo ERROR:
> +echo %% pass-xpass-fail-xfail-skip-error %%
>  END
>  
>  chmod a+x *.t
> @@ -139,25 +141,28 @@ for vpath in : false; do
>    cat pass-fail.log
>    cat pass4-skip.log
>    cat pass3-skip2-xfail.log
> -  cat pass-xpass-fail-xfail-skip.log
> +  cat pass-xpass-fail-xfail-skip-error.log
>    # For debugging.
> -  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
> +  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
>  
>    test `grep -c '^PASS:' stdout` -eq $expected_pass
>    test `grep -c '^FAIL:' stdout` -eq $expected_fail
>    test `grep -c '^XPASS:' stdout` -eq $expected_xpass
>    test `grep -c '^XFAIL:' stdout` -eq $expected_xfail
>    test `grep -c '^SKIP:' stdout` -eq $expected_skip
> +  test `grep -c '^ERROR:' stdout` -eq $expected_error
>  
> -  grep  '^PASS: pass-xpass-fail-xfail-skip.t\, testcase 1' stdout
> -  grep  '^FAIL: pass-xpass-fail-xfail-skip\.t, testcase 2' stdout
> -  grep '^XFAIL: pass-xpass-fail-xfail-skip\.t, testcase 3' stdout
> -  grep '^XPASS: pass-xpass-fail-xfail-skip\.t, testcase 4' stdout
> -  grep  '^SKIP: pass-xpass-fail-xfail-skip\.t, testcase 5' stdout
> +  tst=pass-xpass-fail-xfail-skip-error
> +  grep  "^PASS: $tst\.t, testcase 1" stdout
> +  grep  "^FAIL: $tst\.t, testcase 2" stdout
> +  grep "^XFAIL: $tst\.t, testcase 3" stdout
> +  grep "^XPASS: $tst\.t, testcase 4" stdout
> +  grep  "^SKIP: $tst\.t, testcase 5" stdout
> +  grep "^ERROR: $tst\.t, testcase 6" stdout
>  
>    # Check testsuite summary printed on console.
>    sed -e 's/[()]/ /g' -e 's/^/ /' stdout > t
> -  grep ' 6 of 18 ' t
> +  grep ' 7 of 19 ' t
>    grep ' 1 unexpected pass' t
>    grep ' 4 test.* not run' t
>  
> @@ -170,7 +175,7 @@ for vpath in : false; do
>    grep '%% fail %%' test-suite.log
>    grep '%% fail2 %%' test-suite.log
>    grep '%% pass-fail %%' test-suite.log
> -  grep '%% pass-xpass-fail-xfail-skip %%' test-suite.log
> +  grep '%% pass-xpass-fail-xfail-skip-error %%' test-suite.log
>    test `grep -c '%% ' test-suite.log` -eq 4
>  
>    TESTS='pass.t pass3-skip2-xfail.t' $MAKE -e check >stdout \
> @@ -178,11 +183,11 @@ for vpath in : false; do
>    cat test-suite.log
>    cat stdout
>    # For debugging.
> -  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP)' stdout
> +  $EGREP '(PASS|FAIL|XPASS|XFAIL|SKIP|ERROR)' stdout
>    test `grep -c '^PASS:' stdout` -eq 4
>    test `grep -c '^SKIP:' stdout` -eq 2
>    test `grep -c '^XFAIL:' stdout` -eq 1
> -  $EGREP '^(FAIL|XPASS)' stdout && Exit 1
> +  $EGREP '^(FAIL|XPASS|ERROR)' stdout && Exit 1
>  
>    cd $srcdir
>  
> diff --git a/tests/test-driver-custom-no-html.test 
> b/tests/test-driver-custom-no-html.test
> index 9aceae6..683659b 100755
> --- a/tests/test-driver-custom-no-html.test
> +++ b/tests/test-driver-custom-no-html.test
> @@ -62,6 +62,6 @@ $FGREP 'dummy title' test-suite.log
>  # Sanity check: trying to produce HTML output should fail.
>  $MAKE check-html >output 2>&1 && { cat output; Exit 1; }
>  cat output
> -$EGREP 'SEVERE|ERROR' output
> +$FGREP SEVERE output
>  
>  :
> diff --git a/tests/test-driver-end-test-results.test 
> b/tests/test-driver-end-test-results.test
> index 97c1f29..03af51b 100755
> --- a/tests/test-driver-end-test-results.test
> +++ b/tests/test-driver-end-test-results.test
> @@ -57,6 +57,7 @@ PASS: b.test
>  :test-result:PASS
>  :test-result:END
>  :test-result:FAIL
> +:test-result:ERROR
>  END
>  
>  cat > c.test <<END
> diff --git a/tests/test-driver-global-log.test 
> b/tests/test-driver-global-log.test
> index cdaaff6..35b6fe1 100755
> --- a/tests/test-driver-global-log.test
> +++ b/tests/test-driver-global-log.test
> @@ -44,12 +44,13 @@ END
>  echo SKIP: skip.test > skip.test
>  echo FAIL: fail.test > fail.test
>  echo XPASS: xpass.test > xpass.test
> +echo ERROR: error.test > error.test
>  echo :test-result: PASS > fake-pass.test
>  echo "$tab $tab$tab" > empty.test
>  
>  cat > Makefile.am << 'END'
>  TEST_LOG_DRIVER = ./dummy-driver
> -TESTS = pass.test skip.test fail.test xfail.test xpass.test \
> +TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
>          fake-pass.test empty.test
>  END
>  
> @@ -82,6 +83,7 @@ cat test-suite.log
>  grep '^SKIP: skip\.test$' test-suite.log
>  grep '^FAIL: fail.test$' test-suite.log
>  grep '^XPASS: xpass.test$' test-suite.log
> +grep '^ERROR: error.test$' test-suite.log
>  grep '^:test-result: PASS$' test-suite.log
>  grep "^$tab $tab$tab$" test-suite.log
>  $EGREP 'not seen' test-suite.log && Exit 1
> diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test
> index 2ae75b9..f9482b1 100755
> --- a/tests/test-driver-recheck.test
> +++ b/tests/test-driver-recheck.test
> @@ -54,7 +54,7 @@ END
>    rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
>  done
>  
> -for R in FAIL XPASS UNKNOWN; do
> +for R in FAIL XPASS ERROR UNKNOWN; do
>    echo $R: > $R-1
>    echo $R:foo > $R-2
>    echo $R: bar baz > $R-3
> diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver
> index f06eb1c..113e158 100644
> --- a/tests/trivial-test-driver
> +++ b/tests/trivial-test-driver
> @@ -20,9 +20,10 @@
>  # test results per test script.
>  #
>  # The exit status of the wrapped script is ignored.  Lines in its stdout
> -# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS' or `SKIP'
> -# count as a test case result with the obviously-corresponding outcome.
> -# Every other line is ignored for what concerns the testsuite outcome.
> +# and stderr beginning with `PASS', `FAIL', `XFAIL', `XPASS', `SKIP' or
> +# `ERROR' count as a test case result with the obviously-corresponding
> +# outcome.  Every other line is ignored for what concerns the testsuite
> +# outcome.
>  #
>  # This script is used at least by the `driver-custom-multitest*.test'
>  # tests.
> @@ -61,10 +62,10 @@ tmp_res=$log_file-res.tmp
>    : > $tmp_res
>    while read line; do
>      case $line in
> -      PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*)
> +      PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*|ERROR:*)
>          i=`expr $i + 1`
>          result=`LC_ALL=C expr "$line" : '\([A-Z]*\):.*'`
> -        case $result in FAIL|XPASS) st=1;; esac
> +        case $result in FAIL|XPASS|ERROR) st=1;; esac
>          # Output testcase result to console.
>          echo "$result: $test_name, testcase $i"
>          # Register testcase outcome for the log file.