Re: [PATCH] {maint} tests: new subroutines for test skipping/failing

[Stefano Lattarini]

Hello Ralf.

On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Wed, Jan 19, 2011 at 08:23:20PM CET:
> > On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> > >   AM_TESTS_ENVIRONMENT = foo=1
> > >   TESTS_ENVIRONMENT = foo=2
> > > 
> > > the
> > >   $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) ./test
> > > 
> > > will let Solaris sh pass foo=1 to the test, but bash will pass foo=2.
> > >
> > Oh joy, I hadn't thought about this YAPI (Yet Another Portability Issue).
> 
> Well, Posix even leaves this undefined IIRC.
> 
> > > So I'm afraid that at the very least, AM_TESTS_ENVIRONMENT cannot have
> > > the same semantics (just set variables, no export, no semi-colon at end)
> > > as TESTS_ENVIRONMENT.  So if the developer needs to use `foo=1; export
> > > foo' anyway, there is not so much point in having separated *TESTS_SETUP
> > > variables any more.  Or so I think.
> > >
> > I must agree now.  Oh well, no big deal once TESTS_SETUP and AM_TESTS_SETUP
> > are in place.
> 
> Well, what I was trying to say is: what use is TESTS_SETUP if we expand
> to
>   $(AM_TESTS_SETUP) $(TESTS_SETUP) $(TESTS_ENVIRONMENT)
> 
> ?  It can simply be merged into TESTS_ENVIRONMENT; there is no point in
> keeping them separate.
> 
> And then, we can rename AM_TESTS_SETUP to AM_TESTS_ENVIRONMENT if we
> like, or not (consistency vs. green bikeshed question).
> 
> I'll leave it up to the person who writes the documentation with
> detailed explanation of intended semantics.
> 
Attached is my attempt (I'm not sure the new documentation is organized
in the best possible way, though).  Tested with "make html pdf".

BTW, should this patch (or an amndned version of it) go into maint or
into master?

Thanks,
   Stefano

From 7d0da6a3f08f0133f571abf1b1e4d942369aa9a6 Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Mon, 24 Jan 2011 11:34:32 +0100
Subject: [PATCH] check: new developer-reserved AM_TESTS_SETUP variable

For reference, see the discussion at:
<http://lists.gnu.org/archive/html/automake-patches/2011-01/msg00213.html>

* lib/am/check.am [%?PARALLEL_TESTS%] (am__check_pre): Pass also
$(AM_TESTS_SETUP).  Comments updated, and some typos fixed.
* doc/automake.texi (Simple Tests using parallel-tests): Document
AM_TESTS_SETUP.  Reorder some of the existing documentation a bit.
* tests/parallel-tests-am_tests_setup.test: New test.

From a suggestion by Ralf Wildenhues.
---
 ChangeLog                                |   12 ++++
 doc/automake.texi                        |   47 +++++++++++----
 lib/Automake/tests/Makefile.in           |    9 ++-
 lib/am/check.am                          |    9 ++-
 tests/Makefile.am                        |    1 +
 tests/Makefile.in                        |   10 ++-
 tests/parallel-tests-am_tests_setup.test |   97 ++++++++++++++++++++++++++++++
 7 files changed, 165 insertions(+), 20 deletions(-)
 create mode 100755 tests/parallel-tests-am_tests_setup.test

diff --git a/ChangeLog b/ChangeLog
index c632b44..66d7c3d 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,15 @@
+2011-01-24  Stefano Lattarini  <address@hidden>
+
+       check: new developer-reserved AM_TESTS_SETUP variable
+       For reference, see the discussion at:
+       
<http://lists.gnu.org/archive/html/automake-patches/2011-01/msg00213.html>
+       * lib/am/check.am [%?PARALLEL_TESTS%] (am__check_pre): Pass also
+       $(AM_TESTS_SETUP).  Comments updated, and some typos fixed.
+       * doc/automake.texi (Simple Tests using parallel-tests): Document
+       AM_TESTS_SETUP.  Reorder some of the existing documentation a bit.
+       * tests/parallel-tests-am_tests_setup.test: New test.
+       From a suggestion by Ralf Wildenhues.
+
 2011-01-23  Stefano Lattarini  <address@hidden>
 
        coverage: test semantics of "dummy" per-target flags
diff --git a/doc/automake.texi b/doc/automake.texi
index 3d6a7b0..e5a1c2c 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8648,9 +8648,11 @@ allows to specify inter-test dependencies, lazy reruns 
of tests that
 have not completed in a prior run, summary and verbose output in
 @samp{RST} (reStructuredText) and @samp{HTML} format, and hard errors
 for exceptional failures.  Similar to the simple test driver,
address@hidden, @code{AM_COLOR_TESTS}, @code{XFAIL_TESTS}, and
+ @code{AM_COLOR_TESTS}, @code{XFAIL_TESTS}, and
 the @code{check_*} variables are honored, and the environment variable
address@hidden is set during test execution.
address@hidden is set during test execution. Also, @code{TESTS_ENVIRONMENT}
+is still honored, but is complemented by a new developer-reserved variable
address@hidden (described below).
 
 This test driver is still experimental and may undergo changes in order
 to satisfy additional portability requirements.
@@ -8664,6 +8666,16 @@ depends upon log files created for each single test 
program listed in
 @code{TESTS}, which in turn contain all output produced by the
 corresponding tests.
 
address@hidden VERBOSE
+As with the simple driver above, by default one status line is printed
+per completed test, and a short summary after the suite has completed.
+However, standard output and standard error of the test are redirected
+to a per-test log file, so that parallel execution does not produce
+intermingled output.  The output from failed tests is collected in the
address@hidden file.  If the variable @samp{VERBOSE} is set, this
+file is output after the summary.  For best results, the tests should be
+verbose by default now.
+
 @vindex TEST_EXTENSIONS
 @vindex TEST_LOGS
 Each log file is created when the corresponding test has completed.
@@ -8712,15 +8724,28 @@ and @samp{./wrapper-script -d baz} to produce 
@file{foo.log},
 @samp{TESTS_ENVIRONMENT} variable is still expanded before the driver,
 but should be reserved for the user.
 
address@hidden VERBOSE
-As with the simple driver above, by default one status line is printed
-per completed test, and a short summary after the suite has completed.
-However, standard output and standard error of the test are redirected
-to a per-test log file, so that parallel execution does not produce
-intermingled output.  The output from failed tests is collected in the
address@hidden file.  If the variable @samp{VERBOSE} is set, this
-file is output after the summary.  For best results, the tests should be
-verbose by default now.
address@hidden AM_TESTS_SETUP
+The @code{AM_TESTS_SETUP} variable can be used to run initialization code
+and set environment variables for the tests' runs.  The user can still
+employ the @code{TESTS_ENVIRONMENT} variable to override settings from
address@hidden  Note that, for implementation reasons, if the
address@hidden variable is set, its contents @emph{must} be
+terminated by a semicolon.
+
address@hidden
+# The tests below are expected to use the file descriptor passed
+# in the environment variable 'warn_fileno' to print warnings
+# (e.g., about skipped and failed tests).  If this variable were
+# to be set to `2' (i.e. default stderr), the warnings would be
+# redirected by the automake parallel-tests driver into the .log
+# files.  But the AM_TESTS_SETUP definition below will cause the
+# reasons for skip/failure to be printed to the console instead.
+# The user can still override this by setting TESTS_ENVIRONMENT
+# to e.g. `warn_fileno=2' at make runtime, which will cause the
+# warnings to be sent to the .log files again.
+TESTS = test1.sh test2.sh ...
+AM_TESTS_SETUP = exec 9>&2; warn_fileno=9; export warn_fileno;
address@hidden example
 
 @trindex mostlyclean
 @trindex check-html
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index b4940db..bea7a8d 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -121,9 +121,11 @@ am__text_box = $(AWK) '{                           \
 am__sh_e_setup = case $$- in *e*) set +e;; esac
 # To be inserted before the command running the test.  Creates the
 # directory for the log if needed.  Stores in $dir the directory
-# containing $f, in $tst the test, in $log the log, and passes
-# TESTS_ENVIRONMENT.  Save and restore TERM around use of
-# TESTS_ENVIRONMENT, in case that unsets it.
+# containing $f, in $tst the test, in $log the log.  Executes the
+# developer- defined test setup AM_TESTS_SETUP (if any), and passes
+# TESTS_ENVIRONMENT.  Saves and restores TERM around uses of
+# TESTS_ENVIRONMENT and AM_TESTS_SETUP, in case any of them unsets
+# it.
 am__check_pre = \
 $(am__sh_e_setup);                                     \
 $(am__vpath_adj_setup) $(am__vpath_adj)                        \
@@ -138,6 +140,7 @@ if test -f "./$$f"; then dir=./;                    \
 elif test -f "$$f"; then dir=;                         \
 else dir="$(srcdir)/"; fi;                             \
 tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM;           \
+$(AM_TESTS_SETUP)                                      \
 $(TESTS_ENVIRONMENT)
 # To be appended to the command running the test.  Handle the stdout
 # and stderr redirection, and catch the exit status.
diff --git a/lib/am/check.am b/lib/am/check.am
index 4d10ce9..1827e01 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -93,9 +93,11 @@ am__sh_e_setup = case $$- in *e*) set +e;; esac
 
 # To be inserted before the command running the test.  Creates the
 # directory for the log if needed.  Stores in $dir the directory
-# containing $f, in $tst the test, in $log the log, and passes
-# TESTS_ENVIRONMENT.  Save and restore TERM around use of
-# TESTS_ENVIRONMENT, in case that unsets it.
+# containing $f, in $tst the test, in $log the log.  Executes the
+# developer- defined test setup AM_TESTS_SETUP (if any), and passes
+# TESTS_ENVIRONMENT.  Saves and restores TERM around uses of
+# TESTS_ENVIRONMENT and AM_TESTS_SETUP, in case any of them unsets
+# it.
 am__check_pre =                                                \
 $(am__sh_e_setup);                                     \
 $(am__vpath_adj_setup) $(am__vpath_adj)                        \
@@ -110,6 +112,7 @@ if test -f "./$$f"; then dir=./;                    \
 elif test -f "$$f"; then dir=;                         \
 else dir="$(srcdir)/"; fi;                             \
 tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM;           \
+$(AM_TESTS_SETUP)                                      \
 $(TESTS_ENVIRONMENT)
 
 # To be appended to the command running the test.  Handle the stdout
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 78bacf2..728520b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -567,6 +567,7 @@ parallel-tests7.test \
 parallel-tests8.test \
 parallel-tests9.test \
 parallel-tests10.test \
+parallel-tests-am_tests_setup.test \
 parallel-tests-unreadable-log.test \
 parse.test \
 percent.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 18cb337..0b832b2 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -124,9 +124,11 @@ am__text_box = $(AWK) '{                           \
 am__sh_e_setup = case $$- in *e*) set +e;; esac
 # To be inserted before the command running the test.  Creates the
 # directory for the log if needed.  Stores in $dir the directory
-# containing $f, in $tst the test, in $log the log, and passes
-# TESTS_ENVIRONMENT.  Save and restore TERM around use of
-# TESTS_ENVIRONMENT, in case that unsets it.
+# containing $f, in $tst the test, in $log the log.  Executes the
+# developer- defined test setup AM_TESTS_SETUP (if any), and passes
+# TESTS_ENVIRONMENT.  Saves and restores TERM around uses of
+# TESTS_ENVIRONMENT and AM_TESTS_SETUP, in case any of them unsets
+# it.
 am__check_pre = \
 $(am__sh_e_setup);                                     \
 $(am__vpath_adj_setup) $(am__vpath_adj)                        \
@@ -141,6 +143,7 @@ if test -f "./$$f"; then dir=./;                    \
 elif test -f "$$f"; then dir=;                         \
 else dir="$(srcdir)/"; fi;                             \
 tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM;           \
+$(AM_TESTS_SETUP)                                      \
 $(TESTS_ENVIRONMENT)
 # To be appended to the command running the test.  Handle the stdout
 # and stderr redirection, and catch the exit status.
@@ -834,6 +837,7 @@ parallel-tests7.test \
 parallel-tests8.test \
 parallel-tests9.test \
 parallel-tests10.test \
+parallel-tests-am_tests_setup.test \
 parallel-tests-unreadable-log.test \
 parse.test \
 percent.test \
diff --git a/tests/parallel-tests-am_tests_setup.test 
b/tests/parallel-tests-am_tests_setup.test
new file mode 100755
index 0000000..bf0d0c6
--- /dev/null
+++ b/tests/parallel-tests-am_tests_setup.test
@@ -0,0 +1,97 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# parallel-tests: check AM_TESTS_SETUP support, and its interaction
+# with TESTS_ENVIRONMENT.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_PROG_MKDIR_P
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .sh .test
+TESTS = foo.test bar.sh
+SH_LOG_COMPILER = sh
+AM_TESTS_SETUP = $(MKDIR_P) quux.dir; exec 9>&2; fd=9; export fd;
+EXTRA_DIST = $(TESTS)
+END
+
+cat > foo.test << 'END'
+#! /bin/sh
+echo "FOO='$FOO'"
+echo "BAR='$BAR'"
+test x"$FOO" = x"$BAR"
+END
+chmod a+x foo.test
+
+cat > bar.sh << 'END'
+: ${fd=2}
+eval "echo 'diagnostic message from test bar' >&${fd}"
+END
+
+minicheck ()
+{
+  cat foo.log
+  cat bar.log
+  test -d quux.dir
+}
+
+miniclean ()
+{
+  rmdir quux.dir
+  rm -f foo.log bar.log test-suite.log
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+FOO=1; export FOO
+
+TESTS_ENVIRONMENT='BAR=1' $MAKE check || { cat test-suite.log; Exit 1; }
+minicheck
+miniclean
+TESTS_ENVIRONMENT='BAR=2' $MAKE check && { cat test-suite.log; Exit 1; }
+minicheck
+miniclean
+
+BAR=$FOO; export BAR
+
+$MAKE check 2>stderr || { cat stderr >&2; Exit1; }
+cat stderr >&2
+minicheck
+grep 'diagnostic message from test bar' stderr
+grep 'diagnostic message from test bar' bar.log && Exit 1
+miniclean
+
+TESTS_ENVIRONMENT='fd=2' $MAKE check 2>stderr || { cat stderr >&2; Exit1; }
+cat stderr >&2
+minicheck
+grep 'diagnostic message from test bar' stderr && Exit 1
+grep 'diagnostic message from test bar' bar.log
+miniclean
+
+$MAKE distcheck
+
+:
-- 
1.7.2.3