recheck, recheck-html, check-html

[Ralf Wildenhues]

Hi Akim,

I need to start a new thread, the old one is getting out of hands.

Yeah, well, I managed to mess the recheck recheck-html check-html
targets up.  The only defense that I have is that your versions used
perl and new-style $(command substitution), and the html targets failed
to recreate the log in some occasions, and those glitches and the lack
of supplied tests and weird ordering in the makefile made me think they
were probably all wrong anyway. ;-)

Bugs in my code:

- above three targets fail to update $(check_SCRIPTS) $(check_PROGRAMS).

- recheck* did not run all tests if $(TEST_SUITE_LOG) does not exist.
I'm still not quite sure whether that is the best semantics possible,
but it is strictly better than the current ones of doing nothing upon
  make clean; make recheck

- I'm not yet sure really what to do about tests whose log files are
missing.  recheck them or not?  What do you think?  After a series such
as
  make clean
  make check TESTS='some tests'
  make recheck

would you expect only failures of those two to be tested again?
OTOH, if we did that, then this:

  make clean
  make check TESTS='some tests'
  make recheck TESTS='some more tests'

would not run 'more', which seems surprising, too.  So I'm going with
ignoring nonexistent log files for now.

I've pushed this to 'next', committed in your name as you really did
have this fixed in the version you posted, only I messed it up.

I still need to check whether the TEST_LOGS overrides carry through
portably with all non-GNU make implementations.

Cheers,
Ralf

commit 68b7dc1d9413d84179f879eb6720a585039eb8b3
Author: Akim Demaille <address@hidden>
Date:   Mon Apr 6 07:03:16 2009 +0200

    parallel-tests: redo check-html, recheck, recheck-html.
    
    * lib/am/check.am (recheck, recheck-am): Remove.
    (recheck-TESTS): Rename to ...
    (recheck): ... this and rewrite, factored ...
    (recheck-html): ... with this rule.  Pass TEST_LOGS rather than
    RECHECK_LOGS to `check' and `check-html', respectively, to avoid
    running outdated tests.  Invoking the public macros ensures
    `check_SCRIPTS' etc. are created in time.  Do not output errors
    for tests that were not run yet.  If the testsuite has not run
    at all, run all tests.
    (check-html): Run `check' target, not `check-TESTS', to ensure
    `check_SCRIPTS' etc. are created in time.
    (.PHONY, .MAKE, AM_RECURSIVE_TARGETS): Adjust contents.
    * tests/parallel-tests2.test: Expose the check-html and
    recheck-html issues.
    * tests/parallel-tests9.test: Expose the recheck issues.
    Bugs in previous version pointed out by Akim, who already had
    them fixed in his original version.

diff --git a/lib/am/check.am b/lib/am/check.am
index f5153e3..67ed8cd 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -239,23 +239,7 @@ check-TESTS:
        fi;                                                             \
        $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) $$set_logs
 
-## Rerun all FAILed or XPASSed tests (as well as all whose logs are out
-## of date or do not exist).
-recheck-TESTS:
-       @list='$(TEST_LOGS)';                                           \
-       logs=`for f in $$list; do                                       \
-               if read line < $$f; then                                \
-                 case $$line in FAIL*|XPASS*) echo $$f;; esac;         \
-               else echo $$f; fi;                                      \
-             done | tr '\012\015' '  '`;                               \
-       $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"
-
-recheck-am: recheck-TESTS
-recheck: recheck-am
-.PHONY: recheck recheck-am recheck-TESTS
-.MAKE: recheck-am
-
-AM_RECURSIVE_TARGETS += check recheck
+AM_RECURSIVE_TARGETS += check
 
 ## -------------- ##
 ## Produce HTML.  ##
@@ -275,19 +259,49 @@ AM_RECURSIVE_TARGETS += check recheck
        $$R2H $< >address@hidden
        @mv address@hidden $@
 
-# Be sure to run check-TESTS first, and then to convert the result.
-# Beware of concurrent executions.  And expect check-TESTS to fail.
-check-html recheck-html:
-       @target=`echo $@ | sed 's/-html$$//'`;                  \
-       if $(MAKE) $(AM_MAKEFLAGS) $$target-TESTS; then         \
+# Be sure to run check first, and then to convert the result.
+# Beware of concurrent executions.  Run "check" not "check-TESTS", as
+# check-SCRIPTS and other dependencies are rebuilt by the former only.
+# And expect check to fail.
+check-html:
+       @if $(MAKE) $(AM_MAKEFLAGS) check; then                 \
          rv=0; else rv=$$?;                                    \
        fi;                                                     \
        $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4;   \
        exit $$rv
-.PHONY: check-html recheck-html
-.MAKE: check-html recheck-html
 
-AM_RECURSIVE_TARGETS += check-html recheck-html
+.PHONY: check-html
+.MAKE: check-html
+
+AM_RECURSIVE_TARGETS += check-html
+
+## -------------------- ##
+## Rechecking failures. ##
+## -------------------- ##
+
+## Rerun all FAILed or XPASSed tests (as well as all whose logs are out
+## of date or do not exist).
+recheck recheck-html:
+       @target=`echo $@ | sed 's,^re,,'`;                              \
+       list='$(TEST_LOGS)';                                            \
+## If the test suite has not been run yet, then run it in full
+       if test -f $(TEST_SUITE_LOG); then                              \
+         list=`for f in $$list; do                                     \
+                 test -f $$f || continue;                              \
+                 if read line < $$f; then                              \
+                   case $$line in FAIL*|XPASS*) echo $$f;; esac;       \
+                 else echo $$f; fi;                                    \
+               done | tr '\012\015' '  '`;                             \
+         args=TEST_LOGS=$$list;                                        \
+         $(MAKE) $(AM_MAKEFLAGS) $$target TEST_LOGS="$$list";          \
+       else                                                            \
+         $(MAKE) $(AM_MAKEFLAGS) $$target;                             \
+       fi
+
+.PHONY: recheck recheck-html
+.MAKE: recheck recheck-html
+
+AM_RECURSIVE_TARGETS += recheck recheck-html
 
 else !%?PARALLEL_TESTS%
 
diff --git a/tests/parallel-tests2.test b/tests/parallel-tests2.test
index f9e2a3c..3331a0c 100755
--- a/tests/parallel-tests2.test
+++ b/tests/parallel-tests2.test
@@ -30,11 +30,16 @@ END
 cat > Makefile.am << 'END'
 TEST_SUITE_LOG = mylog.log
 TESTS = foo.test bar.test baz.test
+check_SCRIPTS = bla
+bla:
+       echo bla > $@
+CLEANFILES = bla
 END
 
 cat >>foo.test <<'END'
 #! /bin/sh
 echo "this is $0"
+test -f bla || exit 1
 exit 0
 END
 cat >>bar.test <<'END'
@@ -69,4 +74,15 @@ rm -f mylog.html
 env TESTS=foo.test $MAKE -e recheck-html >stdout || { cat stdout; Exit 1; }
 cat stdout
 test -f mylog.html
+
+# check-html and recheck-html should cause check_SCRIPTS to be created,
+# and recheck-html should rerun all tests if check has not been run.
+$MAKE clean
+env TESTS=foo.test $MAKE -e check-html
+test -f bla
+$MAKE clean
+env TESTS=foo.test $MAKE -e recheck-html
+test -f bla
+test -f foo.log
+test -f mylog.html
 :
diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
index 7c935d9..a61df60 100755
--- a/tests/parallel-tests9.test
+++ b/tests/parallel-tests9.test
@@ -28,11 +28,16 @@ END
 cat > Makefile.am << 'END'
 TEST_SUITE_LOG = mylog.log
 TESTS = foo.test bar.test baz.test
+check_SCRIPTS = bla
+CLEANFILES = bla
+bla:
+       echo bla > $@
 END
 
 cat >>foo.test <<'END'
 #! /bin/sh
 echo "this is $0"
+test -f bla || exit 1
 exit 0
 END
 cat >>bar.test <<'END'
@@ -67,4 +72,13 @@ $MAKE recheck >stdout && { cat stdout; Exit 1; }
 cat stdout
 grep foo.test stdout || Exit 1
 
+# Ensure that recheck builds check_SCRIPTS, and that
+# recheck reruns all tests if check has not been run.
+$MAKE clean
+$MAKE recheck && Exit 1
+test -f bla
+test -f foo.log
+test -f bar.log
+test -f baz.log
+test -f mylog.log
 :