bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures

[Karl Berry]

Hi Glenn,

    The example is meant to be such that the test X in root and SUBDIR
    are tests that would be run when not explicitly specifying tests.

Yep, I got that.

    So its to be able to run any test that would be run when not
    explicitly specifying tests.

Clearly yes, it would be desirable in principle to be able to name tests
anything you want, in any directory you want, and then be able to
specify exactly the individual tests you want to run, regardless of
names or (sub)directories. I'm just not sure how feasible it is to
implement.

    I think the way to support this is to have TESTS be a list of test
    paths relative to the root makefile.  

Isn't it already that way?

    So in the example above, I would
    specify TESTS="X SUBDIR/X".

To run a specific test in automake now, what I came up with is 
make check TESTS=t/sometest.sh

Although the manual says it can be overridden in the environment
(Parallel Test Harness node):
  env TESTS="foo.test bar.test" make -e check
this has never worked for me. (Neither foo.sh or t/foo.sh.) I have not
tried to discern what I was doing wrong. (I feel pretty sure it was me
doing something wrong and not Automake, at least.)

I realize the case you're concerned about is when both ./sometest.sh and
t/sometest.sh are present. My gut reaction is that it would be more
reliable to detect that case and complain about it than try to support
it. But ok, never mind, proceeding along the ideal lines ...

    Before running the recursive make in SUBDIR, the TESTS variable
    would be filtered to only include tests with a relative path of the
    sub directory.  Also, the root makefile would not run tests that
    have a directory component to the path.

It sounds sensible to me, but regrettably, I just don't have a great
overall grip on the code, having come to the implementation side of
Automake rather late. Jim or Paul would have better things to say.
Unfortunately I think they are busy with other things (which is why I'm
here in the first place ... anyway ...).

    I would like to get a rough idea of if this would be an approved of
    approach so that I or someone else could take a stab at implementing it
    with the hopes of the changes being accepted.

In general, the top priority for any change in Automake is to not break
existing practice. If you can devise a change that, for starters, still
passes all the existing Automake tests, then I surmise there's a good
shot.

As you may infer from my TESTS= stuff above, exactly what invariants
have to be maintained to keep compatibility in this case is not clear to
me :(.

(Also, as one would expect, the change has to be comprehensible and
maintainable, but that's nothing special to Automake.)

Regarding your previous patch, it actually seemed pretty plausible to me
(modulo EXEEXT and other such, as you noted; not a big deal), except
that it's a shame to lose the typo checking. There are over a thousand
tests in Automake, and no doubt that many or more in other packages. So
typos are inevitable. Seems like some sort of separate check for them
would need to be available, going down that road.

Thanks for following up. I hope you are sufficiently interested to spend
some more time on it. --best, karl.