[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated
|
From: |
Stefano Lattarini |
|
Subject: |
[Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated. v1.11-1118-g5f5b225 |
|
Date: |
Sun, 07 Aug 2011 14:20:32 +0000 |
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".
http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=5f5b225a674cde633e42bb3ca3f767ca95ffe6a7
The branch, tap-testsuite-work has been updated
via 5f5b225a674cde633e42bb3ca3f767ca95ffe6a7 (commit)
via 5c1bb4e240c3538a6bfa7026dfe5f590ac747863 (commit)
via 3f5dcfb065552773b4b0fe295647a9c16363d180 (commit)
via 11aa54925224bb51b85655be1f7d46c6cc0fcf2d (commit)
via 0286308c4923e9b1547811587988fbec73aabf00 (commit)
via 7fb047693f4f820960605a681e6a8f1687dc4a9f (commit)
via 297c8cf72445fd026024b963cd5390f6d7345d3c (commit)
via 45567bd1db489a9745e139dc980a1092624175e7 (commit)
from 273593955a62bbbd7e20b91367c66b6e547d4882 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 5f5b225a674cde633e42bb3ca3f767ca95ffe6a7
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 15:54:26 2011 +0200
test defs: get name of TAP-using tests correctly
This change fixes the names of the temporary directories used
by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir'
rather than `foo.tap.dir'); more importantly, this make the
variable `$destdir' defined in `tests/defs' usable also in the
TAP tests.
($me): Also strip the any `.tap' suffix from the name of the
test.
commit 5c1bb4e240c3538a6bfa7026dfe5f590ac747863
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 15:20:00 2011 +0200
testsuite: TAP tests properly decide when to remove tempdirs
Before this change, the TAP tests in the Automake testsuite were
removing the temporary test directory even when they failed or
were skipped, thus making debugging more difficult.
* tests/tap-functions.sh (incr_tap_count): Removed, superseded
by ...
(incr_): ... this function, which can increment the value of any
variable passed to it.
(result_): Updated to use `incr_' instead of the now-removed
`incr_tap_count_'. Keep count of failures, xfailures, xpasses,
and skips, using ...
($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ...
these new variables.
* tests/defs (trap): Try to use their values to decide whether
the temporary directory being used by the test script should be
removed or not. Other code reorganizations. And move the code
for the removal of the temporary directory out to ...
(rm_rf_): ... this new subroutine.
(Main code): Use that instead of duplicating the code.
commit 3f5dcfb065552773b4b0fe295647a9c16363d180
Author: Stefano Lattarini <address@hidden>
Date: Sat Aug 6 22:42:24 2011 +0200
testsuite: improve and refactor our custom TAP shell library
This change is mostly done in preparation for the next one.
* tests/tap-functions.sh (result_with_directive_): Merge into ...
(result_): ... this function, which has been greatly extended and
improved in various ways.
(ok_, not_ok_, skip_, xfail_, xpass_): Adjust.
Adjust comments. Remove an extra empty line.
* tests/instspc.tap: Use the new improved interface of `result_'.
* tests/tap-bad-prog.tap: Likewise.
commit 11aa54925224bb51b85655be1f7d46c6cc0fcf2d
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 11:17:18 2011 +0200
testsuite: update "user interface" description in tests/README
* tests/README (Subsection "Interpretation"): Mention the "ERROR"
outcome.
(Subsection "Getting details from failures"): Don't tell that the
test scripts can be run directly, this is not 100% true; anymore;
instead ...
(Subsection "About the tests"): ... use this new subsection to
explain how to run the tests (either simple or TAP-based) by hand.
(Subsection "Supported shells"): Extend to refer to TAP tests too.
commit 0286308c4923e9b1547811587988fbec73aabf00
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 09:42:48 2011 +0200
testsuite: TAP tests can have early plan now
* tests/defs: When using TAP, allow the user to specify the number
of tests through the `$planned' variable.
(trap): Do not print the auto-generated plan if an explicit one
has already been printed.
* tests/defs-static.in: Complain and bail out if the `$planned'
variable is exported in the environment. Reformat the code to
facilitate future additions.
* tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'.
* tests/self-check-env-sanitize.test: Update.
* tests/ac-output-old.tap: Give an explicit TAP plan, by defining
`$planned' just before including `./defs'.
* tests/depmode.tap: Likewise.
* tests/automake-cmdline.tap: Likewise.
* tests/tap-bad-prog.tap: Likewise.
* tests/instspc.tap: Likewise, and related adjustments.
* tests/add-missing.tap: Make it explicit that we have no a-priori
plan, and tell why.
commit 7fb047693f4f820960605a681e6a8f1687dc4a9f
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 11:31:13 2011 +0200
testsuite: run autogenerated tests with $(LOG_COMPILER) too
This change enhances consistency in the testsuite.
* tests/Makefile.am: Some cosmetic reordering, to keep the
definition of FOO_LOG_COMPILER near to the place where `.foo'
is added to $(TEST_EXTENSIONS).
(PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with
`$(LOG_COMPILER)', not with `$(SHELL)'.
(SHTST_LOG_COMPILER): Likewise.
commit 297c8cf72445fd026024b963cd5390f6d7345d3c
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 7 11:22:11 2011 +0200
testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER)
With this change, it will be easier to override, at make time, the
shell used to run the tests.
* tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'.
(TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'.
(TAP_LOG_COMPILER): Likewise.
commit 45567bd1db489a9745e139dc980a1092624175e7
Author: Stefano Lattarini <address@hidden>
Date: Sat Aug 6 22:29:17 2011 +0200
testsuite: run TAP-based tests explicitly with $(SHELL)
* tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'.
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 103 ++++++++++++++++++++++++++++++++++++
tests/Makefile.am | 14 +++--
tests/Makefile.in | 19 ++++---
tests/README | 37 +++++++++++--
tests/ac-output-old.tap | 1 +
tests/add-missing.tap | 1 +
tests/automake-cmdline.tap | 2 +
tests/defs | 77 ++++++++++++++++++---------
tests/defs-static.in | 13 ++++-
tests/depmod.tap | 4 ++
tests/instspc.tap | 14 ++++--
tests/self-check-env-sanitize.test | 1 +
tests/tap-bad-prog.tap | 4 +-
tests/tap-functions.sh | 96 ++++++++++++++++++++++-----------
14 files changed, 301 insertions(+), 85 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 1366010..4bff03b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,106 @@
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ test defs: get name of TAP-using tests correctly
+ This change fixes the names of the temporary directories used
+ by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir'
+ rather than `foo.tap.dir'); more importantly, this make the
+ variable `$destdir' defined in `tests/defs' usable also in the
+ TAP tests.
+ ($me): Also strip the any `.tap' suffix from the name of the
+ test.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: TAP tests properly decide when to remove tempdirs
+ Before this change, the TAP tests in the Automake testsuite were
+ removing the temporary test directory even when they failed or
+ were skipped, thus making debugging more difficult.
+ * tests/tap-functions.sh (incr_tap_count): Removed, superseded
+ by ...
+ (incr_): ... this function, which can increment the value of any
+ variable passed to it.
+ (result_): Updated to use `incr_' instead of the now-removed
+ `incr_tap_count_'. Keep count of failures, xfailures, xpasses,
+ and skips, using ...
+ ($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ...
+ these new variables.
+ * tests/defs (trap): Try to use their values to decide whether
+ the temporary directory being used by the test script should be
+ removed or not. Other code reorganizations. And move the code
+ for the removal of the temporary directory out to ...
+ (rm_rf_): ... this new subroutine.
+ (Main code): Use that instead of duplicating the code.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: improve and refactor our custom TAP shell library
+ This change is mostly done in preparation for the next one.
+ * tests/tap-functions.sh (result_with_directive_): Merge into ...
+ (result_): ... this function, which has been greatly extended and
+ improved in various ways.
+ (ok_, not_ok_, skip_, xfail_, xpass_): Adjust.
+ Adjust comments. Remove an extra empty line.
+ * tests/instspc.tap: Use the new improved interface of `result_'.
+ * tests/tap-bad-prog.tap: Likewise.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: update "user interface" description in tests/README
+ * tests/README (Subsection "Interpretation"): Mention the "ERROR"
+ outcome.
+ (Subsection "Getting details from failures"): Don't tell that the
+ test scripts can be run directly, this is not 100% true; anymore;
+ instead ...
+ (Subsection "About the tests"): ... use this new subsection to
+ explain how to run the tests (either simple or TAP-based) by hand.
+ (Subsection "Supported shells"): Extend to refer to TAP tests too.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: TAP tests can have early plan now
+ * tests/defs: When using TAP, allow the user to specify the number
+ of tests through the `$planned' variable.
+ (trap): Do not print the auto-generated plan if an explicit one
+ has already been printed.
+ * tests/defs-static.in: Complain and bail out if the `$planned'
+ variable is exported in the environment. Reformat the code to
+ facilitate future additions.
+ * tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'.
+ * tests/self-check-env-sanitize.test: Update.
+ * tests/ac-output-old.tap: Give an explicit TAP plan, by defining
+ `$planned' just before including `./defs'.
+ * tests/depmode.tap: Likewise.
+ * tests/automake-cmdline.tap: Likewise.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/instspc.tap: Likewise, and related adjustments.
+ * tests/add-missing.tap: Make it explicit that we have no a-priori
+ plan, and tell why.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: run autogenerated tests with $(LOG_COMPILER) too
+ This change enhances consistency in the testsuite.
+ * tests/Makefile.am: Some cosmetic reordering, to keep the
+ definition of FOO_LOG_COMPILER near to the place where `.foo'
+ is added to $(TEST_EXTENSIONS).
+ (PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with
+ `$(LOG_COMPILER)', not with `$(SHELL)'.
+ (SHTST_LOG_COMPILER): Likewise.
+
+2011-08-07 Stefano Lattarini <address@hidden>
+
+ testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER)
+ With this change, it will be easier to override, at make time, the
+ shell used to run the tests.
+ * tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'.
+ (TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'.
+ (TAP_LOG_COMPILER): Likewise.
+
+2011-08-06 Stefano Lattarini <address@hidden>
+
+ testsuite: run TAP-based tests explicitly with $(SHELL)
+ * tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'.
+
2011-08-06 Stefano Lattarini <address@hidden>
tap driver: handle signals received by the tests being run
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 1865bbd..76d14fd 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -16,7 +16,13 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
+# Run the tests with the shell detected at configure time.
+LOG_COMPILER = $(SHELL)
+
TEST_EXTENSIONS = .test .tap
+TEST_LOG_COMPILER = $(LOG_COMPILER)
+TAP_LOG_COMPILER = $(LOG_COMPILER)
+
TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
TAP_LOG_DRIVER_FLAGS = --merge --comments
## Avoid spurious TAP diagnostic. We use an awful indirection with a
@@ -32,9 +38,6 @@ TAP_LOG_DRIVER_FLAGS += --diagnostic-string `printf
'\043%%\043\n'`
MAINTAINERCLEANFILES =
EXTRA_DIST = ChangeLog-old
-# Run the tests with the shell detected at configure time.
-TEST_LOG_COMPILER = $(SHELL)
-
XFAIL_TESTS = \
all.test \
auxdir2.test \
@@ -56,7 +59,7 @@ $(srcdir)/parallel-tests.am: gen-parallel-tests Makefile.am
EXTRA_DIST += gen-parallel-tests
TEST_EXTENSIONS += .ptest
-PTEST_LOG_COMPILER = $(SHELL) $(srcdir)/parallel-tests.sh
+PTEST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/parallel-tests.sh
EXTRA_DIST += parallel-tests.sh
# All `*-p.ptest' tests work by sourcing the script `parallel-tests.sh'.
@@ -73,7 +76,7 @@ $(srcdir)/config-shell-tests.am: gen-config-shell-tests
Makefile.am
EXTRA_DIST += gen-config-shell-tests
TEST_EXTENSIONS += .shtst
-SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
+SHTST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/config-shell-tests.sh
EXTRA_DIST += config-shell-tests.sh
# All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
@@ -88,6 +91,7 @@ $(config_shell_tests):
AM_TESTS_ENVIRONMENT = \
test x"$$me" = x || unset me; \
test x"$$required" = x || unset required; \
+ test x"$$planned" = x || unset planned; \
test x"$$use_tap" = x || unset use_tap; \
test x"$$parallel_tests" = x || unset parallel_tests; \
test x"$$test_prefer_config_shell" || unset test_prefer_config_shell; \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index e28d6ed..cd30b56 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -289,7 +289,12 @@ target_alias = @target_alias@
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
+
+# Run the tests with the shell detected at configure time.
+LOG_COMPILER = $(SHELL)
TEST_EXTENSIONS = .test .tap .ptest .shtst
+TEST_LOG_COMPILER = $(LOG_COMPILER)
+TAP_LOG_COMPILER = $(LOG_COMPILER)
TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
TAP_LOG_DRIVER_FLAGS = --merge --comments --diagnostic-string `printf \
'\043%%\043\n'`
@@ -299,9 +304,6 @@ EXTRA_DIST = ChangeLog-old gen-parallel-tests
parallel-tests.sh \
$(handwritten_tests) tap-functions.sh plain-functions.sh \
trivial-test-driver testsuite-summary-checks.sh \
extract-testsuite-summary tap-setup.sh tap-summary-aux.sh
-
-# Run the tests with the shell detected at configure time.
-TEST_LOG_COMPILER = $(SHELL)
XFAIL_TESTS = \
all.test \
auxdir2.test \
@@ -324,13 +326,13 @@ parallel_tests = backcompat5-p.ptest
check-exported-srcdir-p.ptest \
comment9-p.ptest dejagnu-p.ptest exeext4-p.ptest \
maken3-p.ptest maken4-p.ptest posixsubst-tests-p.ptest \
tests-environment-p.ptest
-PTEST_LOG_COMPILER = $(SHELL) $(srcdir)/parallel-tests.sh
+PTEST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/parallel-tests.sh
config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \
compile3-w.shtst compile4-w.shtst compile5-w.shtst \
compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \
mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \
missing3-w.shtst missing5-w.shtst mkinst3-w.shtst
-SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
+SHTST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/config-shell-tests.sh
# Some testsuite-influential variables should be overridable from the
# test scripts, but not from the environment.
@@ -339,9 +341,10 @@ SHTST_LOG_COMPILER = $(SHELL)
$(srcdir)/config-shell-tests.sh
# we are already running them under it explicitly in our setup (see e.g.
# the definition of TEST_LOG_COMPILER above).
AM_TESTS_ENVIRONMENT = test x"$$me" = x || unset me; test \
- x"$$required" = x || unset required; test x"$$use_tap" = x || \
- unset use_tap; test x"$$parallel_tests" = x || unset \
- parallel_tests; test x"$$test_prefer_config_shell" || unset \
+ x"$$required" = x || unset required; test x"$$planned" = x || \
+ unset planned; test x"$$use_tap" = x || unset use_tap; test \
+ x"$$parallel_tests" = x || unset parallel_tests; test \
+ x"$$test_prefer_config_shell" || unset \
test_prefer_config_shell; test x"$$original_AUTOMAKE" = x || \
unset original_AUTOMAKE; test x"$$original_ACLOCAL" = x || \
unset original_ACLOCAL; AM_TESTS_REEXEC=no; export \
diff --git a/tests/README b/tests/README
index b3ff506..8b8da14 100644
--- a/tests/README
+++ b/tests/README
@@ -56,15 +56,15 @@ Interpretation
Other:
SKIP - skipped tests (third party tools not available)
+ ERROR - some unexpected error condition
Getting details from failures
-----------------------------
- Each test is a shell script. In a non-VPATH build you can run the
- tests directly, they will be verbose. By default, verbose output of
- a test foo.test is retained in the log file foo.log. A summary log
- is created in the file test-suite.log.
+ By default, verbose output of a test `foo.test' or `foo.tap' is
+ retained in the log file `foo.log'. A summary log is created in
+ the file `test-suite.log'.
You can limit the set of files using the TESTS variable, and enable
detailed test output at the end of the test run with the VERBOSE
@@ -72,6 +72,30 @@ Getting details from failures
env VERBOSE=x TESTS='first.test second.test ...' make -e check
+ You can also run the tests by hand, as explained in the next subsection.
+
+
+About the tests
+---------------
+
+ There are two kinds of tests in the Automake testsuite (both implemented
+ as shell scripts). The scripts with the `.test' suffix are "simple"
+ tests, their outcome completely determined by their exit status. Those
+ with the `.tap' suffix use the TAP protocol. If you want to run a test
+ by hand, you can do so directly if it is a simple test:
+
+ ./nogzip.test
+
+ (it will be verbose by default), while if it is a TAP test you can pass
+ it to your preferred TAP runner, as in e.g.:
+
+ prove --verbose --merge ./add-missing.tap
+
+ The tests can also be run directly in a VPATH build, as with:
+
+ /path/to/srcdir/tests/nogzip.test
+ prove --verbose --merge /path/to/srcdir/tests/add-missing.tap
+
Supported shells
----------------
@@ -82,11 +106,12 @@ Supported shells
`/path/to/another/sh', the user must use:
AM_TESTS_REEXEC=no /path/to/another/sh ./foo.test
+ AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./bar.tap
to run a test directly, and:
- make check TEST_LOG_COMPILER=/path/to/sh (GNU make)
- TEST_LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
+ make check LOG_COMPILER=/path/to/sh (GNU make)
+ LOG_COMPILER=/path/to/sh make -e check (non-GNU make)
to run the test(s) through the makefile test driver.
diff --git a/tests/ac-output-old.tap b/tests/ac-output-old.tap
index 74bbba3..201e517 100755
--- a/tests/ac-output-old.tap
+++ b/tests/ac-output-old.tap
@@ -19,6 +19,7 @@
# line breaking in the use of AC_OUTPUT. Synthesised by a bunch
# of older tests (referenced below).
+planned=22
. ./defs || Exit 1
rm -f configure.in depcomp # Not required.
diff --git a/tests/add-missing.tap b/tests/add-missing.tap
index 4c6a624..cbf6e57 100755
--- a/tests/add-missing.tap
+++ b/tests/add-missing.tap
@@ -19,6 +19,7 @@
# the files) it's supposed to, and that these files are symlinked by
# default, but copied if the `--copy' option is used.
+planned=unknown # Counting it would be too much data-dependent.
. ./defs || Exit 1
build_aux=build-aux
diff --git a/tests/automake-cmdline.tap b/tests/automake-cmdline.tap
index 592c73b..1c53a6d 100755
--- a/tests/automake-cmdline.tap
+++ b/tests/automake-cmdline.tap
@@ -15,6 +15,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Test Automake's command-line options.
+
+planned=17
. ./defs || Exit 1
# Usage: bad_cmdline DESCRIPTION REGEX-FOR-STDERR [ARGS-FOR-AUTOMAKE...]
diff --git a/tests/defs b/tests/defs
index c468d5f..25cfb93 100644
--- a/tests/defs
+++ b/tests/defs
@@ -43,7 +43,7 @@ test -f ./defs-static || {
# be done carefully, and *before* including ./defs).
if test -z "$me"; then
# Guard against failure to spawn sed (seen on MSYS), or empty $argv0.
- me=`echo "$argv0" | sed -e 's,.*[\\/],,;s/\.test$//'` \
+ me=`echo "$argv0" | sed -e 's,.*[\\/],,;s/\.test$//;s/\.tap$//;'` \
&& test -n "$me" \
|| { echo "$argv0: failed to define \$me" >&2; exit 99; }
fi
@@ -322,6 +322,20 @@ seq_ ()
done
}
+# rm_rf_ [FILES OR DIRECTORIES ...]
+# ---------------------------------
+# Recursively remove the given files or directory, also handling the case
+# of non-writable subdirectories.
+rm_rf_ ()
+{
+ test $# -gt 0 || return 0
+ # Ignore failures in find, we are only interested in failures of the
+ # final rm.
+ find "$@" -type d ! -perm -700 -exec chmod u+rwx {} \; || :
+ rm -rf "$@"
+}
+
+
# count_test_results total=N pass=N fail=N xpass=N xfail=N skip=N error=N
# -----------------------------------------------------------------------
# Check that a testsuite run driven by the parallel-tests harness has
@@ -430,6 +444,11 @@ xsi_shell_code='
## required by them. ##
## ----------------------------------------------------------- ##
+# The TAP plan must be either the fist or last line of output.
+if test $using_tap = yes && test ${planned-unknown} != unknown; then
+ plan_ "$planned"
+fi
+
# Print it here, so that the user will see it also if the test
# will be skipped due to some tool missing in $PATH itself.
echo "$PATH"
@@ -797,10 +816,7 @@ distdir=$me-1.0
# this directory will be kept, to facilitate debugging.
testSubDir=$me.dir
-test ! -d $testSubDir || {
- find $testSubDir -type d ! -perm -700 -exec chmod u+rwx {} ";"
- rm -rf $testSubDir
-}
+test ! -d $testSubDir || rm_rf_ $testSubDir
mkdir $testSubDir
cd ./$testSubDir
@@ -809,29 +825,40 @@ if test "$sh_errexit_works" = yes; then
trap 'exit_status=$?
set +e
cd "$testbuilddir"
- # This is to ensure that a test script does give a SKIP outcome just
- # because a command in it happens to exit with status 77. This
- # behaviour, while from time to time useful to developers, is not
- # meant to be enabled by default, as it could cause spurious failures
- # in the wild. Thus it will be enabled only when the variable
- # "am_explicit_skips" is set to a "true" value.
- case $am_explicit_skips in
- [yY]|[yY]es|1)
- if test $exit_status -eq 77 && test $am__test_skipped != yes; then
- exit_status=78
- fi
- ;;
- esac
- case $exit_status,$keep_testdirs in
- 0,)
- find $testSubDir -type d ! -perm -700 -exec chmod u+rwx {} ";"
- rm -rf $testSubDir
- ;;
- esac
+ if test -n "$keep_testdirs"; then
+ keep_testdirs=yes
+ else
+ keep_testdirs=no
+ fi
+ if test $using_tap = yes; then
+ if test $have_tap_plan_ != yes; then
+ late_plan_
+ fi
+ test $exit_status -eq 0 \
+ && test $tap_xfail_count_ -eq 0 \
+ && test $tap_skip_count_ -eq 0 \
+ && test $tap_bad_count_ -eq 0 \
+ || keep_testdirs=yes
+ else
+ # This is to ensure that a test script does give a SKIP outcome just
+ # because a command in it happens to exit with status 77. This
+ # behaviour, while from time to time useful to developers, is not
+ # meant to be enabled by default, as it could cause spurious failures
+ # in the wild. Thus it will be enabled only when the variable
+ # "am_explicit_skips" is set to a "true" value.
+ case $am_explicit_skips in
+ [yY]|[yY]es|1)
+ if test $exit_status -eq 77 && test $am__test_skipped != yes; then
+ echo "$me: implicit skip turned into failure"
+ exit_status=78
+ fi;;
+ esac
+ test $exit_status -eq 0 || keep_testdirs=yes
+ fi
+ test $keep_testdirs = no && rm_rf_ $testSubDir
set +x
test "$signal" != 0 && echo "$me: caught signal $signal"
echo "$me: exit $exit_status"
- test $using_tap = yes && late_plan_
exit $exit_status
' 0
for signal in 1 2 13 15; do
diff --git a/tests/defs-static.in b/tests/defs-static.in
index 407060d..73eaad1 100644
--- a/tests/defs-static.in
+++ b/tests/defs-static.in
@@ -70,9 +70,16 @@ fi
# Check that the environment is properly sanitized.
# Having variables exported to the empty string is OK, since our code
# treats such variables as if they were unset.
-for var in me required use_tap parallel_tests test_prefer_config_shell \
- original_AUTOMAKE original_ACLOCAL;
-do
+for var in \
+ me \
+ required \
+ planned \
+ use_tap \
+ parallel_tests \
+ test_prefer_config_shell \
+ original_AUTOMAKE \
+ original_ACLOCAL \
+; do
if eval "test x\"\$$var\" != x" && env | grep "^$var=" >/dev/null; then
echo "$argv0: variable \`$var' is set in the environment:" \
"this is unsafe" >&2
diff --git a/tests/depmod.tap b/tests/depmod.tap
index 4470fdd..85bd972 100755
--- a/tests/depmod.tap
+++ b/tests/depmod.tap
@@ -51,6 +51,10 @@
# which caused make to die with an error like:
# "sub/.deps/subfoo.Po:3: *** missing separator. Stop."
+# Per each valid depmode three kinds of VPATH, and per each of them
+# a test on "./configure" and one on "make & remake"; so there are
+# 17 * 3 * 2 = 102.
+planned=102
required=cc
. ./defs || Exit 99
diff --git a/tests/instspc.tap b/tests/instspc.tap
index 64540d8..f421fc7 100755
--- a/tests/instspc.tap
+++ b/tests/instspc.tap
@@ -19,6 +19,7 @@
# Original report from James Amundson about file names with spaces.
# Other characters added by Paul Eggert.
+planned=94 # Two tests per "problematic string".
. ./defs || Exit 99
# Usage: is_in_list ITEM [LIST...]
@@ -222,7 +223,7 @@ for test_name in $test_names_list; do
# characters in file names.
mkdir "./$test_string" || \
- skip_row_ 2 -r "mkdir failed" "$test_name"
+ skip_ -r "mkdir failed" "$test_name"
# Where are the "weird" characters going to be used, in $(builddir)
# or in $(DESTDIR)? They are always going to be used in $(prefix)
@@ -256,11 +257,16 @@ for test_name in $test_names_list; do
&& DESTDIR="$dest" file="./$test_string" $MAKE -e test-inst \
|| r='not ok'
- tap_text="$test_name in ${where}dir"
+ description="$test_name in ${where}dir"
if expected_to_fail "$where" "$test_name"; then
- tap_text="$tap_text # TODO long-standing limitation"
+ directive=TODO
+ reason="long-standing limitation"
+ else
+ directive=
+ reason=
fi
- result_ "$r" "$tap_text" # Test case outcome is here.
+ # Test case outcome is here.
+ result_ "$r" -D "$directive" -r "$reason" -- "$description"
cd "$ocwd" || fatal_ "cannot chdir back to test directory"
diff --git a/tests/self-check-env-sanitize.test
b/tests/self-check-env-sanitize.test
index e2d8f2a..84c6b9f 100755
--- a/tests/self-check-env-sanitize.test
+++ b/tests/self-check-env-sanitize.test
@@ -27,6 +27,7 @@ vars='
me
parallel_tests
required
+ planned
use_tap
test_prefer_config_shell
original_AUTOMAKE
diff --git a/tests/tap-bad-prog.tap b/tests/tap-bad-prog.tap
index 2dee975..086b38e 100755
--- a/tests/tap-bad-prog.tap
+++ b/tests/tap-bad-prog.tap
@@ -18,6 +18,7 @@
# - missing, unreadable, or not-executable test scripts cause proper
# error reports
+planned=5
parallel_tests=yes
. ./defs || Exit 1
@@ -81,6 +82,7 @@ then
else
r='not ok'
fi
-result_ "$r" 'no spurious results # TODO still get "missing plan"'
+result_ "$r" --directive TODO --reason 'still get "missing plan"' \
+ 'no spurious results'
:
diff --git a/tests/tap-functions.sh b/tests/tap-functions.sh
index d87a32c..d685c18 100644
--- a/tests/tap-functions.sh
+++ b/tests/tap-functions.sh
@@ -24,15 +24,22 @@
# The count of the TAP test results seen so far.
tap_count_=0
+# The count of skipped tests.
+tap_skip_count_=0
+# The count of tests that experienced an expected failure.
+tap_xfail_count_=0
+# The count of tests with unexpected outcomes (i.e., failed and xpassed).
+tap_bad_count_=0
# The first "test -n" tries to avoid extra forks when possible.
if test -n "${ZSH_VERSION}${BASH_VERSION}" \
|| (eval 'test $((1 + 1)) = 2') >/dev/null 2>&1
then
- # Use of 'eval' needed to protect dumber shells from parsing errors.
- eval 'incr_tap_count_ () { tap_count_=$(($tap_count_ + 1)); }'
+ # Outer use of 'eval' needed to protect dumber shells from parsing
+ # errors.
+ eval 'incr_ () { eval "$1=\$((\${$1} + 1))"; }'
else
- incr_tap_count_ () { tap_count_=`expr $tap_count_ + 1`; }
+ incr_ () { eval "$1=\`expr \${$1} + 1\`"; }
fi
# plan_ NUMBER-OF-PLANNED-TESTS
@@ -83,29 +90,53 @@ warn_ ()
diag_ "WARNING:" ${1-"(unknown warning)"} ${1+"$@"}
}
-# result_ RESULT [DESCRIPTION...]
-# -------------------------------
-# Report a test case with the given result.
+# result_ RESULT [-D DIRECTIVE] [-r REASON] [--] [DESCRIPTION...]
+# ---------------------------------------------------------------
+# Report a test case with the given RESULT (valid values are "ok" and
+# "not ok") and the given DESCRIPTION (if any). If DIRECTIVE is given
+# and non-empty (valid values being "TODO" and "SKIP"), it will be
+# reported too, with the REASON (if given) appended.
result_ ()
{
- incr_tap_count_
- tap_result_=$1; shift
- echo "$tap_result_" $tap_count_ ${1+"- $*"}
-}
-
-# tap_with_directive_ RESULT DIRECTIVE [-r REASON] [DESCRIPTION...]
-# -----------------------------------------------------------------
-# Write a tap result with the given directive (can be "TODO" or "SKIP").
-# The REASON, if given is appended after the directive. This function is
-# for internal use only.
-result_with_directive_ ()
-{
- incr_tap_count_
+ set +x # Don't pollute the log files.
+ test $# -gt 0 || bailout_ "result_: missing argument"
tap_result_=$1; shift
- tap_directive_=$1; shift
- case $1 in -r) tap_reason_=" $2" shift 2;; *) tap_reason_="";; esac
- echo "$tap_result_" $tap_count_ ${1+"- $*"} \
- "# ${tap_directive_}${tap_reason_}"
+ case $tap_result_ in
+ "ok"|"not ok") ;;
+ *) bailout_ "result_: invalid result '$tap_result'" ;;
+ esac
+ tap_directive_= tap_reason_=
+ while test $# -gt 0; do
+ case $1 in
+ -D|--directive) tap_directive_=$2; shift;;
+ -r|--reason) tap_reason_=$2; shift;;
+ --) shift; break;;
+ -*) bailout_ "result_: invalid option '$1'";;
+ *) break;;
+ esac
+ shift
+ done
+ case $tap_directive_ in
+ ""|TODO|SKIP) ;;
+ *) bailout_ "result_: invalid directive '$directive_'" ;;
+ esac
+ incr_ tap_count_
+ case $tap_result_,$tap_directive_ in
+ ok,) ;; # Passed.
+ not\ ok,TODO) incr_ tap_xfail_count_;; # Expected failure.
+ not\ ok,*|ok,TODO) incr_ tap_bad_count_ ;; # Failed or xpassed.
+ ok,SKIP) incr_ tap_skip_count_ ;; # Skipped.
+ *) bailout_ "internal error in 'result_'";; # Can't happen.
+ esac
+ tap_text_="$tap_result_ $tap_count_"
+ if test x"$*" != x; then
+ tap_text_="$tap_text_ - $*"
+ fi
+ if test x"$tap_directive_" != x; then
+ tap_text_="$tap_text_ # $tap_directive_"${tap_reason_:+" $tap_reason_"}
+ fi
+ printf '%s\n' "$tap_text_"
+ set -x # Restore shell xtraces.
}
# ok_ [DESCRIPTION...]
@@ -113,7 +144,7 @@ result_with_directive_ ()
# Report a successful test.
ok_ ()
{
- result_ 'ok' ${1+"$@"}
+ result_ 'ok' -- ${1+"$@"}
}
# not_ok_ [DESCRIPTION...]
@@ -121,20 +152,20 @@ ok_ ()
# Report a failed test.
not_ok_ ()
{
- result_ 'not ok' ${1+"$@"}
+ result_ 'not ok' -- ${1+"$@"}
}
-# skip_ [-r REASON] [DESCRIPTION...]
-# ----------------------------------
+# skip_ [-r REASON] [--] [DESCRIPTION...]
+# ---------------------------------------
# Report a skipped test. If the `-r' option is present, its argument is
# give as the reason of the skip.
skip_ ()
{
- result_with_directive_ 'ok' SKIP ${1+"$@"}
+ result_ 'ok' -D SKIP ${1+"$@"}
}
-# skip_row_ COUNT [-r REASON] [DESCRIPTION...]
-# --------------------------------------------
+# skip_row_ COUNT [-r REASON] [--] [DESCRIPTION...]
+# -------------------------------------------------
# Report a COUNT of skipped test, with the given reason and descriptions
# (if any). Useful to avoid cascade failures in case a fair number of
# tests depend on an earlier one that failed.
@@ -150,7 +181,7 @@ skip_row_ ()
# argument is give as the reason why the failure is expected.
xfail_ ()
{
- result_with_directive_ 'not ok' TODO ${1+"$@"}
+ result_ 'not ok' -D TODO ${1+"$@"}
}
# xpass_ [-r REASON] [DESCRIPTION...]
@@ -159,7 +190,7 @@ xfail_ ()
# argument is give as the reason why the failure is expected.
xpass_ ()
{
- result_with_directive_ 'ok' TODO ${1+"$@"}
+ result_ 'ok' -D TODO ${1+"$@"}
}
# skip_all_ [REASON ...]
@@ -231,7 +262,6 @@ command_not_ok_ ()
fi
}
-
# reset_test_count_ COUNT
# -----------------------
# Reset the count of the TAP test results seen so far to COUNT.
hooks/post-receive
--
GNU Automake
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated. v1.11-1118-g5f5b225,
Stefano Lattarini <=