Re: recheck, recheck-html, check-html

[Ralf Wildenhues]

Hi Akim,

* Akim Demaille wrote on Thu, Apr 09, 2009 at 09:49:43AM CEST:
> Le 6 avr. 09 à 07:52, Ralf Wildenhues a écrit :
>
> I'm sorry for being so unresponsive, I hope I'm not causing some delays 
> in the release of Automake!

Not really.  I have a nonempty list of other buglets to fix still,
although luckily the number of regressions is very low.  Mostly trying
to avoid enshrining any ugly spots in the API that we'd then have to
support for a long time.  And of course I want vala support added.

When things are done, I will, in the end, decide as I see fit and
release the thing.  But I will wait another week to see if licensing
issues have sorted themselves out till then, and coordinate with Eric
about an Autoconf release.

>> Bugs in my code:
>>
>> - above three targets fail to update $(check_SCRIPTS) $ 
>> (check_PROGRAMS).
>
> But these guys are less important than they were.  Now dependencies  
> matter, and these black-box coarse-grain dependencies are no good.

I agree.

> I'm not asking to drop their support, but maybe the doc should
> emphasize that correct dependencies are more important.

Do you think it should be emphasize even more?  Would you like to
propose a patch against the manual?

>  Actually, is there any reason to have check_PROGRAMS in the new
>  framework?

I really would like to let the old setup work more or less OOTB with the
parallel-tests driver.  Consistency is Good, and allowing incremental
improvements can help acceptance.

For example, the gnulib test suite has tests scripts that run some
check_PROGRAMS.  Listing these dependencies manually would be work
(of course the idea is that gnulib-tool will do it automatically
eventually), but a parallel test suite would be beneficial even without
that work being done.

>> - recheck* did not run all tests if $(TEST_SUITE_LOG) does not exist.
>> I'm still not quite sure whether that is the best semantics possible,
>> but it is strictly better than the current ones of doing nothing upon
>>  make clean; make recheck
>
> I think it should do nothing.
[...]

> I see "recheck" as a simple means to define some TESTS, so I see  
> "recheck TESTS=..." as incorrect.  Therefore, in this case I would say  
> the behavior is "undefined" in the documentation, and have the  
> implementation do whatever is the simplest.
>
> I use recheck in very short edit/test cycles, and actually, I realize  
> that it is a burden that it depends on TEST_SUITE_LOGS.  I think it  
> would be more useful if it did not, and work only on the log files.   
> That's because it does happen (at least to me) that I see some tests  
> fail, and interrupt the test suite immediately.  So I do have *.log  
> files, but I do not have a test-suite.log.  In which case I would find  
> it more useful to rerun just the tests that failed.
>
> I agree this is not what my implementation did, but now that you  
> pinpoint parts of its behavior, that's what I would do.

Thanks for explaining this.  I agree with you now.  I've now pushed this
patch to the 'next' branch to implement it.

Cheers,
Ralf

    parallel-tests: `clean recheck' should not rerun all tests.
    
    * lib/am/check.am (recheck, recheck-html): Do not rerun all
    tests if `$(TEST_SUITE_LOG)' does not exist.
    * tests/parallel-tests2.test: Adjust recheck test.
    * tests/parallel-tests9.test: Adjust recheck-html test.
    * NEWS: Update.
    Suggestion by Akim Demaille.

diff --git a/NEWS b/NEWS
index cf01518..64756fe 100644
--- a/NEWS
+++ b/NEWS
@@ -14,6 +14,9 @@ New in 1.10c:
 
     to the rules.
 
+  - The new `parallel-tests' targets `recheck' and `recheck-html' will not run
+    any tests that have not run yet.
+
 Bugs fixed in 1.10c:
 
 * Long standing bugs:
diff --git a/lib/am/check.am b/lib/am/check.am
index 90cb033..f0917a0 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -279,23 +279,17 @@ AM_RECURSIVE_TARGETS += check-html
 ## Rechecking failures. ##
 ## -------------------- ##
 
-## Rerun all FAILed or XPASSed tests (as well as all whose logs are out
-## of date or do not exist).
+## Rerun all FAILed or XPASSed tests.
 recheck recheck-html:
        @target=`echo $@ | sed 's,^re,,'`;                              \
        list='$(TEST_LOGS)';                                            \
-## If the test suite has not been run yet, then run it in full
-       if test -f $(TEST_SUITE_LOG); then                              \
-         list=`for f in $$list; do                                     \
-                 test -f $$f || continue;                              \
-                 if read line < $$f; then                              \
-                   case $$line in FAIL*|XPASS*) echo $$f;; esac;       \
-                 else echo $$f; fi;                                    \
-               done | tr '\012\015' '  '`;                             \
-         $(MAKE) $(AM_MAKEFLAGS) $$target TEST_LOGS="$$list";          \
-       else                                                            \
-         $(MAKE) $(AM_MAKEFLAGS) $$target;                             \
-       fi
+       list=`for f in $$list; do                                       \
+               test -f $$f || continue;                                \
+               if read line < $$f; then                                \
+                 case $$line in FAIL*|XPASS*) echo $$f;; esac;         \
+               else echo $$f; fi;                                      \
+             done | tr '\012\015' '  '`;                               \
+       $(MAKE) $(AM_MAKEFLAGS) $$target TEST_LOGS="$$list"
 
 .PHONY: recheck recheck-html
 .MAKE: recheck recheck-html
diff --git a/tests/parallel-tests2.test b/tests/parallel-tests2.test
index 3331a0c..2a5fc6f 100755
--- a/tests/parallel-tests2.test
+++ b/tests/parallel-tests2.test
@@ -76,13 +76,13 @@ cat stdout
 test -f mylog.html
 
 # check-html and recheck-html should cause check_SCRIPTS to be created,
-# and recheck-html should rerun all tests if check has not been run.
+# and recheck-html should rerun no tests if check has not been run.
 $MAKE clean
 env TESTS=foo.test $MAKE -e check-html
 test -f bla
 $MAKE clean
 env TESTS=foo.test $MAKE -e recheck-html
 test -f bla
-test -f foo.log
+test ! -f foo.log
 test -f mylog.html
 :
diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
index a61df60..6678920 100755
--- a/tests/parallel-tests9.test
+++ b/tests/parallel-tests9.test
@@ -73,12 +73,12 @@ cat stdout
 grep foo.test stdout || Exit 1
 
 # Ensure that recheck builds check_SCRIPTS, and that
-# recheck reruns all tests if check has not been run.
+# recheck reruns nothing if check has not been run.
 $MAKE clean
-$MAKE recheck && Exit 1
+$MAKE recheck
 test -f bla
-test -f foo.log
-test -f bar.log
-test -f baz.log
+test ! -f foo.log
+test ! -f bar.log
+test ! -f baz.log
 test -f mylog.log
 :