Re: parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.

[Ralf Wildenhues]

Salut Akim,

* Akim Demaille wrote on Wed, Apr 01, 2009 at 04:53:28PM CEST:
> Le 31 mars 09 à 01:01, Ralf Wildenhues a écrit :
>> * Akim Demaille wrote on Mon, Mar 30, 2009 at 09:50:59PM CEST:
>>> Le 29 mars 09 à 00:04, Ralf Wildenhues a écrit :
>>>>
>>>> This is what the 'recheck' target does, right?
>>>
>>> Not exactly.  make recheck runs only the tests that failed, while
>>> here, in addition, the tests whose dependencies have changed will be
>>> run  again.  Not to mention test that where not run at all.
>>
>> I don't think your interpretation is right.  Those tests whose
>> dependencies have changed will be rerun in any case, with your
>> recheck, with my recheck, with above.  You cannot prevent it unless
>> you remove those dependency statements from the makefile.
>
> Err, really?  My code reads:

> So I really get the list of logs that were FAIL'd, remove them, and run 
> them as TESTS.  Here I'm using the fact that TESTS accepts *.log in my 
> implementation.  I don't see why logs that passed but are obsoleted would 
> run again.

Got it now.  Your 'recheck' is different from what I added as
'recheck'.  Mine doesn't pass TESTS=... to the sub-make, so also
outdated tests will be rerun.

I will address this (and other issues around this) in a separate post.

>>> I have not understood exactly what you are referring too, but why  
>>> don't we just pass TESTS=$(TESTS) explicitly?
>>
>> Because it can cause command line length issues.  Because variable
>> overriding is inherently fragile: it can subtly change the meaning
>> of the underlying makefile.
>>
>> When another indirection for globbing were added, we'd have to pass
>> $(TESTS) through more than one $(MAKE).  And we'd have to pass
>> $(TEST_LOGS) as well, if we were to allow globbing on that.  And
>> I would like to avoid passing both, that halves our available length
>> limit.
>
> Wow.  What's the worse case for this length?

$ info Automake 'Length Limitations'

after you install recent Automake.  :-)

> How about having another  
> mode to run the test suite, based on a temporary MANITEST file with the 
> list of files to run?  Maybe a __am_tests.mk file and a 'make -f  
> __am_tests.mk check'?  It would have list the dependencies from test- 
> suite.log to the individual logs, and include the main Makefile.  I  
> dunno, maybe I don't make any sense here, or maybe there are  
> complications I have no foreseen.

I would like to avoid going that way for now.  To get around the length
limitations, there are several possibilities.  The one that currently
works with TESTS is to simply split up the testsuite in several
directories and have several Makefile.am files.

Another valid extension would be to have multiple test suites in one
Makefile.am, the way you proposed a couple of times earlier.  I like
that approach better than the MANIFEST file one, because it is more in
line with Automake semantics in general.  Let's look at this in more
detail after 1.11, though.  (So, probably in a couple of years.  ;-)

> Are there non-GNU Makes that support -j?

Yes.  I don't see how that is directly relevant to this question,
though: the parallel-tests driver is supposed to be usable without -j,
too.  And by all kinds of make implementations.

You touch another valid point though: Other make -jN implementations
have ugly bugs in them.  I will post more on this after finishing
testing, and sending bug reports to the BSD camp.  :-)

>>> Or ask the users to add his "precious macros" to AM_MAKEFLAGS?
>>
>> We could do that.  In fact, I intend to add a note to this extent to
>> the Autoconf manual section about `make' portability, but
>> AM_MAKEFLAGS should be mentioned in automake.texi too I guess.  What
>> I don't know  is whether it was intended as user variable or as
>> makefile.am author  realm.
>
> Given it's name, I think it should be userland.  __AM_ whatever is a  
> different beast, but AM_CFLAGS and so forth are definitely userland.

But the most prominent package using AM_MAKEFLAGS, GCC, overrides it
unconditionally in most sub makefiles.

>>> Can't we have it when using GNU Make?  Can't we have some sort of  
>>> #ifdef
>>> HAVE_GNU_MAKE sections in the output Makefile?  I'm not referring to
>>> AM_CONDITIONAL, but to some nice run-time trick.
>>
>> Produce a GNUmakefile in the build directory.  Let this GNUmakefile
>> include the Makefile.  Put all your nice goodies in GNUmakefile.
>>
>> An example of this is what the gnulib module gnumakefile does.
[...]
> Yes, I also have some GNUmakefiles where I want to play that kind of  
> trick.  But this kind of clashes with users that already do.  I was  
> looking for something for Automake, not for the Automake user.

Suggestions welcome!

Cheers,
Ralf