bug#42883: Selecting a subset of tests in recursive make causes unnecessary failures

[Glenn Washburn]

Hi,

I've been working on some automake tests for grub, and ran in to a bug.
I noticed that when specifying a subset of tests to run via the TESTS
environment variable, I could have all tests complete successfully and
yet still have make return an error.

Here's a sequence of commands that should reproduce the error:
   git clone https://git.savannah.gnu.org/git/grub.git
   cd grub
   ./bootstrap
   ./configure
   make
   env TESTS=grub_cmd_echo make -e check VERBOSE=yes

It turns out that make check is being recursively run in the grub-core
directory, as expected.  The TESTS environment variable is the same in
both invocations, also as expected.  However, the test program is not in
the grub-core subdirectory.  So the test is run successfully in the root
make, and not found in the subdirectory, thus generating an error. In
fact, grub has no tests in the grub-core subdirectory, but the harness
is run because its part of automake, even though I do not need or want
it to.

I've attached a patch which fixes this issue by ignoring tests which do
not exist. I'm not sure this patch takes care of everything needed, for
instance EXEEXT, to check for the existence of the test program. Grub
does not use any TEST_EXTENSIONS, so this works.

This will change make it so TESTS which should exist, but do not, will
not cause an error, whereas currently it will.  I'm not sure how this
situation would arise, because make should require the existence as a
dependency.  If there is an error in the TESTS variable (eg. a typo in
a test name), this will not cause an error either, which I deem
undesirable. However, I think this is an acceptable trade-off.

Perhaps a better solution would be to not recursively run make check if
TESTS is defined and run all given tests from the root make. Tests
which are run in recursive make invocations when TESTS is not defined
can be run via TESTS by specifying a path component (eg.
TESTS="grub-core/test-program").  This could be a problem if test
programs assume they are run from the directory in which they reside
(in which case, we could chdir to the directory and run the program).

Any thoughts?
Glenn

fix-recursive-subset-test.patch
Description: Text Data

grub-tests-error.txt
Description: Text document