Re: recheck, recheck-html, check-html

[Akim Demaille]

Le 6 avr. 09 à 07:52, Ralf Wildenhues a écrit :

> Hi Akim,

Hi Ralf,

I'm sorry for being so unresponsive, I hope I'm not causing some  
delays in the release of Automake!

> Bugs in my code:
>
> - above three targets fail to update $(check_SCRIPTS) $ 
> (check_PROGRAMS).

But these guys are less important than they were.  Now dependencies  
matter, and these black-box coarse-grain dependencies are no good.   
I'm not asking to drop their support, but maybe the doc should  
emphasize that correct dependencies are more important.  Actually, is  
there any reason to have check_PROGRAMS in the new framework?


> - recheck* did not run all tests if $(TEST_SUITE_LOG) does not exist.
> I'm still not quite sure whether that is the best semantics possible,
> but it is strictly better than the current ones of doing nothing upon
>  make clean; make recheck

I think it should do nothing.


> - I'm not yet sure really what to do about tests whose log files are
> missing.  recheck them or not?  What do you think?  After a series  
> such
> as
>  make clean
>  make check TESTS='some tests'
>  make recheck
>
> would you expect only failures of those two to be tested again?
> OTOH, if we did that, then this:
>
>  make clean
>  make check TESTS='some tests'
>  make recheck TESTS='some more tests'
>
> would not run 'more', which seems surprising, too.  So I'm going with
> ignoring nonexistent log files for now.

I see "recheck" as a simple means to define some TESTS, so I see  
"recheck TESTS=..." as incorrect.  Therefore, in this case I would say  
the behavior is "undefined" in the documentation, and have the  
implementation do whatever is the simplest.

I use recheck in very short edit/test cycles, and actually, I realize  
that it is a burden that it depends on TEST_SUITE_LOGS.  I think it  
would be more useful if it did not, and work only on the log files.   
That's because it does happen (at least to me) that I see some tests  
fail, and interrupt the test suite immediately.  So I do have *.log  
files, but I do not have a test-suite.log.  In which case I would find  
it more useful to rerun just the tests that failed.

I agree this is not what my implementation did, but now that you  
pinpoint parts of its behavior, that's what I would do.

> diff --git a/lib/am/check.am b/lib/am/check.am
> index f5153e3..67ed8cd 100644
> --- a/lib/am/check.am
> +++ b/lib/am/check.am
> @@ -239,23 +239,7 @@ check-TESTS:
>         fi;                                                             \
>         $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) $$set_logs
>
> -## Rerun all FAILed or XPASSed tests (as well as all whose logs are  
> out
> -## of date or do not exist).
> -recheck-TESTS:
> -       @list='$(TEST_LOGS)';                                           \
> -       logs=`for f in $$list; do                                       \
> -               if read line < $$f; then                             \
> -                 case $$line in FAIL*|XPASS*) echo $$f;; esac;         \
> -               else echo $$f; fi;                                      \
> -             done | tr '\012\015' '  '`;                               \
> -       $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"
> -
> -recheck-am: recheck-TESTS
> -recheck: recheck-am
> -.PHONY: recheck recheck-am recheck-TESTS
> -.MAKE: recheck-am
> -
> -AM_RECURSIVE_TARGETS += check recheck
> +AM_RECURSIVE_TARGETS += check
>
> ## -------------- ##
> ## Produce HTML.  ##
> @@ -275,19 +259,49 @@ AM_RECURSIVE_TARGETS += check recheck
>         $$R2H $< >address@hidden
>         @mv address@hidden $@
>
> -# Be sure to run check-TESTS first, and then to convert the result.
> -# Beware of concurrent executions.  And expect check-TESTS to fail.
> -check-html recheck-html:
> -       @target=`echo $@ | sed 's/-html$$//'`;                  \
> -       if $(MAKE) $(AM_MAKEFLAGS) $$target-TESTS; then         \
> +# Be sure to run check first, and then to convert the result.
> +# Beware of concurrent executions.  Run "check" not "check-TESTS", as
> +# check-SCRIPTS and other dependencies are rebuilt by the former  
> only.
> +# And expect check to fail.
> +check-html:
> +       @if $(MAKE) $(AM_MAKEFLAGS) check; then                 \
>           rv=0; else rv=$$?;                                    \
>         fi;                                                     \
>         $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4;   \
>         exit $$rv
> -.PHONY: check-html recheck-html
> -.MAKE: check-html recheck-html
>
> -AM_RECURSIVE_TARGETS += check-html recheck-html
> +.PHONY: check-html
> +.MAKE: check-html
> +
> +AM_RECURSIVE_TARGETS += check-html
> +
> +## -------------------- ##
> +## Rechecking failures. ##
> +## -------------------- ##
> +
> +## Rerun all FAILed or XPASSed tests (as well as all whose logs are  
> out
> +## of date or do not exist).
> +recheck recheck-html:
> +       @target=`echo $@ | sed 's,^re,,'`;                              \
> +       list='$(TEST_LOGS)';                                            \
> +## If the test suite has not been run yet, then run it in full
> +       if test -f $(TEST_SUITE_LOG); then                              \
> +         list=`for f in $$list; do                                     \
> +                 test -f $$f || continue;                              \
> +                 if read line < $$f; then                           \
> +                   case $$line in FAIL*|XPASS*) echo $$f;; esac;       \
> +                 else echo $$f; fi;                                    \
> +               done | tr '\012\015' '  '`;                             \
> +         args=TEST_LOGS=$$list;                                        \
> +         $(MAKE) $(AM_MAKEFLAGS) $$target TEST_LOGS="$$list";                \
> +       else                                                            \
> +         $(MAKE) $(AM_MAKEFLAGS) $$target;                             \
> +       fi
> +
> +.PHONY: recheck recheck-html
> +.MAKE: recheck recheck-html
> +
> +AM_RECURSIVE_TARGETS += recheck recheck-html
>
> else !%?PARALLEL_TESTS%
>
> diff --git a/tests/parallel-tests2.test b/tests/parallel-tests2.test
> index f9e2a3c..3331a0c 100755
> --- a/tests/parallel-tests2.test
> +++ b/tests/parallel-tests2.test
> @@ -30,11 +30,16 @@ END
> cat > Makefile.am << 'END'
> TEST_SUITE_LOG = mylog.log
> TESTS = foo.test bar.test baz.test
> +check_SCRIPTS = bla
> +bla:
> +       echo bla > $@
> +CLEANFILES = bla
> END
>
> cat >>foo.test <<'END'
> #! /bin/sh
> echo "this is $0"
> +test -f bla || exit 1
> exit 0
> END
> cat >>bar.test <<'END'
> @@ -69,4 +74,15 @@ rm -f mylog.html
> env TESTS=foo.test $MAKE -e recheck-html >stdout || { cat stdout;  
> Exit 1; }
> cat stdout
> test -f mylog.html
> +
> +# check-html and recheck-html should cause check_SCRIPTS to be  
> created,
> +# and recheck-html should rerun all tests if check has not been run.
> +$MAKE clean
> +env TESTS=foo.test $MAKE -e check-html
> +test -f bla
> +$MAKE clean
> +env TESTS=foo.test $MAKE -e recheck-html
> +test -f bla
> +test -f foo.log
> +test -f mylog.html
> :
> diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test
> index 7c935d9..a61df60 100755
> --- a/tests/parallel-tests9.test
> +++ b/tests/parallel-tests9.test
> @@ -28,11 +28,16 @@ END
> cat > Makefile.am << 'END'
> TEST_SUITE_LOG = mylog.log
> TESTS = foo.test bar.test baz.test
> +check_SCRIPTS = bla
> +CLEANFILES = bla
> +bla:
> +       echo bla > $@
> END
>
> cat >>foo.test <<'END'
> #! /bin/sh
> echo "this is $0"
> +test -f bla || exit 1
> exit 0
> END
> cat >>bar.test <<'END'
> @@ -67,4 +72,13 @@ $MAKE recheck >stdout && { cat stdout; Exit 1; }
> cat stdout
> grep foo.test stdout || Exit 1
>
> +# Ensure that recheck builds check_SCRIPTS, and that
> +# recheck reruns all tests if check has not been run.
> +$MAKE clean
> +$MAKE recheck && Exit 1
> +test -f bla
> +test -f foo.log
> +test -f bar.log
> +test -f baz.log
> +test -f mylog.log
> :