Re: Documentation for the parallel-tests driver. [4/4]

[Ralf Wildenhues]

Hello again Akim,

I see that I need to be more verbose in some of my thoughts.

* Akim Demaille wrote on Mon, Oct 20, 2008 at 12:31:33PM CEST:
> Le 18 oct. 08 à 08:22, Ralf Wildenhues a écrit :
>>
>> Which is a nice idea.  I like it.  I even tried to go this way when I
>> first started.  It turned out that a number of purely practical issues
>> needed to be addressed:
>>
>> - if you still want to be able to easily limit the number of tests run
>>  at 'make' run time, you have to go through hoops to ensure that
>>  *_OBJECTS is a direct dependent of *_SOURCES (resp. TESTS),
>
> I don't know what you mean here.  Maybe you are referring to hard errors 
> as you implemented them?

No.  I'm referring to being able to write
  env TESTS="only.t some.t of.t many.t tests.t" make -e check

and having only 5 tests run.  This is important, as it allows much more
fine-grained subsetting than would for example choosing tests by
extension.  For example, I often use something like this:
  cd $srcdir
  set x `grep -li libtool *.test`
  shift
  cd -
  env TESTS="$*" make -e check

This only works well if automake emits something like
  TEST_LOGS = $(TESTS:.test=.log)

or
  am__logs_1 = $(TESTS:.test=.log)
  TEST_LOGS = $(am__logs_1:.exe.log=.log)

which is what I meant with "*_OBJECTS is a direct dependent of *_SOURCES
(resp. TESTS)", and the primary reason I want to forbid users to specify
TEST_LOGS themselves.

And the only thing I was complaining about was that I was having
problems integrating this easily into the normal Automake mechanisms
of registered languages.

This feature, to me, is just as important as being able to specify
different test drivers, essentially because the subsetting mechanism
isn't fixed in advance.

(But I think we can have both.)

>> - *for this language*, the mapping from sources to objects should  
>> *not* be an automake internal detail,
>
> Well, unless you want to expose the concept of "Automake rule", say
>
>       .log.calc = ./calc-tester $(CALC_FLAGS) $@ >$<

No, I definitely don't want to have something like that.

> and process them, we either have to use regular rules, or regular  
> variables.  The rules does not seem so bad to me, the fully flown rule  
> will always be "something before, the command, something afterwards".   
> So it's an Ugly API™, absolutely, but it's effective and can evolve  
> IMHO.

>> - then, I'm quite a fan of backward compatibility, with the aim that
>>  most users would get a working parallel testsuite by simply adding
>>    AUTOMAKE_OPTIONS = parallel-tests
>>
>>  to their Makefile.am.  Cf. testsuite addition that tests just this.
>
> I fail to see what changes here.  By default the test-driver is nul,  
> that is, you just run the test itself.

Yeah, probably that isn't a problem.

>> - In your proposal, it would IMVHO still be necessary to specify the  
>> set of extensions that tests can have (in order to support @substed@
>>  tests at least, and in order to avoid ambiguities and per-target
>>  rules).
>
> I agree that it is nicer if it is Automake that catches these errors,  
> but if it doesn't Make is still here, and it will.  I don't understand  
> what you mean with per-target rules here.

If I have
  TESTS = a.test b.chk c d.sh @substed@

then automake must somehow *know* what the set of log names is.  Is it
  a.log b.log c.log d.log ... (something else we don't know)

or is it
  a.test.log b.chk.log c.log d.sh.log ... (something else we don't know)

or some mixture of both.  See the ambiguity?  automake *must* learn
explicitly about the extensions it is to rip off of the test names,
and for each such extension, it will be able to use a suffix rule
  .chk.log: ; ...

and for each test without a known (or without any) extension, automake
must emit a per-target rule
  c.log: c ; ...

and since automake cannot know what to do with @substed@ tests, either
they all have to have some known extension, or there must be
EXTRA_*_TESTS.  However, EXTRA_*_TESTS conflict with my easily limiting
requirement at the very start of this email.

So, your proposal *needs* to include
- a way of specifying each extension used, and
- a "default" test driver to use for each extension-less test,

in order to be viable.  And we should be able to agree on disallowing
EXTRA_*_TESTS, and requiring known suffixes for @substed@ tests.

> It looks like you're aiming for much more general than what I did/need.  
> Yet losing the genericity I need :(

I think the differences are more in practicality than generality.

Cheers,
Ralf