[PATCH] {master} Extend and improve tests on DejaGnu support.

[Stefano Lattarini]

Another testsuite patch, based off of maint, but to be merged
in master only.

I will wait the customary 72 hours before pushing.

Regards,
   Stefano

-*-*-*-

Extend and improve tests on DejaGnu support.

* tests/dejagnu.test: Do not create useless dummy test script.
Add trailing `:' command.  In heading comments, add reference
to ...
* tests/check12.test: ... this new "semantic" test, covering
concurrent use of dejagnu tests, simple tests and `check-local'
target.
* tests/dejagnu2.test: Make test more reliable, by avoid weak
grepping of make output.  Prefer `cat' over `echo' to append
to configure.in.  Quote literal dots in grep regexps.  Prefer
`grep -c ...' over `grep ... | wc -l'.  Make grepping of
automake stderr slighty stricter.  Add trailing `:' command.
* tests/dejagnu3.test: Prefer `cat' over `echo' to append to
configure.in.  Check stderr of expected-to-fail "make" call.
Remove extra blank lines from Makefile.am.
* tests/dejagnu4.test: Prefer `cat' over `echo' to append to
configure.in.  Prefer `mv -f' over plain `mv' when the target
file already exists.  Avoid extra mkdir calls by creating more
directories at once.  Better use of blank lines.  Check that
the `*.log' and `*.sum' files are created by runtest also when
"make check" fails.
* tests/dejagnu7.test: Prefer `cat' over `echo' to append to
configure.in.  Better use of blank lines.  Add a trailing `:'
command.
* tests/dejagnu6.test: Likewise, and give the dejagnu test a
more descriptive name.
* tests/dejagnu5.test: Likewise.  Also, simply define package
name to `$me' rather than using a non-obvious sed script to
extract it from `AC_INIT', and write the Makefile.am with only
one command.
* tests/Makefile.am (TESTS): Updated.
---
 ChangeLog           |   34 +++++++++
 tests/Makefile.am   |    1 +
 tests/Makefile.in   |    2 +
 tests/check12.test  |  201 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/dejagnu.test  |    7 +-
 tests/dejagnu2.test |   21 +++---
 tests/dejagnu3.test |   17 +++--
 tests/dejagnu4.test |   19 +++--
 tests/dejagnu5.test |   19 +++---
 tests/dejagnu6.test |   16 ++--
 tests/dejagnu7.test |   10 ++--
 11 files changed, 298 insertions(+), 49 deletions(-)
 create mode 100755 tests/check12.test

From 39ab915725c5eb5fe055dd3e09116982b1abcce1 Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Sat, 16 Oct 2010 17:03:03 +0200
Subject: [PATCH] Extend and improve tests on DejaGnu support.

* tests/dejagnu.test: Do not create useless dummy test script.
Add trailing `:' command.  In heading comments, add reference
to ...
* tests/check12.test: ... this new "semantic" test, covering
concurrent use of dejagnu tests, simple tests and `check-local'
target.
* tests/dejagnu2.test: Make test more reliable, by avoid weak
grepping of make output.  Prefer `cat' over `echo' to append
to configure.in.  Quote literal dots in grep regexps.  Prefer
`grep -c ...' over `grep ... | wc -l'.  Make grepping of
automake stderr slighty stricter.  Add trailing `:' command.
* tests/dejagnu3.test: Prefer `cat' over `echo' to append to
configure.in.  Check stderr of expected-to-fail "make" call.
Remove extra blank lines from Makefile.am.
* tests/dejagnu4.test: Prefer `cat' over `echo' to append to
configure.in.  Prefer `mv -f' over plain `mv' when the target
file already exists.  Avoid extra mkdir calls by creating more
directories at once.  Better use of blank lines.  Check that
the `*.log' and `*.sum' files are created by runtest also when
"make check" fails.
* tests/dejagnu7.test: Prefer `cat' over `echo' to append to
configure.in.  Better use of blank lines.  Add a trailing `:'
command.
* tests/dejagnu6.test: Likewise, and give the dejagnu test a
more descriptive name.
* tests/dejagnu5.test: Likewise.  Also, simply define package
name to `$me' rather than using a non-obvious sed script to
extract it from `AC_INIT', and write the Makefile.am with only
one command.
* tests/Makefile.am (TESTS): Updated.
---
 ChangeLog           |   34 +++++++++
 tests/Makefile.am   |    1 +
 tests/Makefile.in   |    2 +
 tests/check12.test  |  201 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/dejagnu.test  |    7 +-
 tests/dejagnu2.test |   21 +++---
 tests/dejagnu3.test |   17 +++--
 tests/dejagnu4.test |   19 +++--
 tests/dejagnu5.test |   19 +++---
 tests/dejagnu6.test |   16 ++--
 tests/dejagnu7.test |   10 ++--
 11 files changed, 298 insertions(+), 49 deletions(-)
 create mode 100755 tests/check12.test

diff --git a/ChangeLog b/ChangeLog
index a356958..4fd15f1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,37 @@
+2010-11-14  Stefano Lattarini  <address@hidden>
+
+       Extend and improve tests on DejaGnu support.
+       * tests/dejagnu.test: Do not create useless dummy test script.
+       Add trailing `:' command.  In heading comments, add reference
+       to ...
+       * tests/check12.test: ... this new "semantic" test, covering
+       concurrent use of dejagnu tests, simple tests and `check-local'
+       target.
+       * tests/dejagnu2.test: Make test more reliable, by avoid weak
+       grepping of make output.  Prefer `cat' over `echo' to append
+       to configure.in.  Quote literal dots in grep regexps.  Prefer
+       `grep -c ...' over `grep ... | wc -l'.  Make grepping of
+       automake stderr slighty stricter.  Add trailing `:' command.
+       * tests/dejagnu3.test: Prefer `cat' over `echo' to append to
+       configure.in.  Check stderr of expected-to-fail "make" call.
+       Remove extra blank lines from Makefile.am.
+       * tests/dejagnu4.test: Prefer `cat' over `echo' to append to
+       configure.in.  Prefer `mv -f' over plain `mv' when the target
+       file already exists.  Avoid extra mkdir calls by creating more
+       directories at once.  Better use of blank lines.  Check that
+       the `*.log' and `*.sum' files are created by runtest also when
+       "make check" fails.
+       * tests/dejagnu7.test: Prefer `cat' over `echo' to append to
+       configure.in.  Better use of blank lines.  Add a trailing `:'
+       command.
+       * tests/dejagnu6.test: Likewise, and give the dejagnu test a
+       more descriptive name.
+       * tests/dejagnu5.test: Likewise.  Also, simply define package
+       name to `$me' rather than using a non-obvious sed script to
+       extract it from `AC_INIT', and write the Makefile.am with only
+       one command.
+       * tests/Makefile.am (TESTS): Updated.
+
 2010-11-07  Stefano Lattarini  <address@hidden>
 
        Fix a bug in variable concatanation with `+='.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index da81c49..9d8cd81 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -133,6 +133,7 @@ check8.test \
 check9.test \
 check10.test \
 check11.test \
+check12.test \
 checkall.test \
 clean.test \
 clean2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index eb461a9..8e14e1e 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -283,6 +283,7 @@ parallel_tests = \
 check-p.test \
 check10-p.test \
 check11-p.test \
+check12-p.test \
 check2-p.test \
 check3-p.test \
 check4-p.test \
@@ -400,6 +401,7 @@ check8.test \
 check9.test \
 check10.test \
 check11.test \
+check12.test \
 checkall.test \
 clean.test \
 clean2.test \
diff --git a/tests/check12.test b/tests/check12.test
new file mode 100755
index 0000000..8be6123
--- /dev/null
+++ b/tests/check12.test
@@ -0,0 +1,201 @@
+#! /bin/sh
+# Copyright (C) 2010 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Test to make sure dejagnu tests, automake-style tests, and check-local
+# target can coexist.
+
+. ./defs || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+AUTOMAKE_OPTIONS =
+EXTRA_DIST =
+CLEANFILES =
+END
+
+## Simple tests.
+
+cat > a.test << 'END'
+#!/bin/sh
+exit ${A_EXIT_STATUS-0}
+END
+cat > b.test << 'END'
+#!/bin/sh
+exit ${B_EXIT_STATUS-0}
+END
+chmod +x a.test b.test
+
+cat >> Makefile.am << 'END'
+TESTS = a.test b.test
+EXTRA_DIST += $(TESTS)
+END
+
+A_EXIT_STATUS=0; export A_EXIT_STATUS
+B_EXIT_STATUS=0; export B_EXIT_STATUS
+
+## DejaGnu tests.
+
+cat >> Makefile.am << 'END'
+AUTOMAKE_OPTIONS += dejagnu
+DEJATOOL = hammer spanner
+AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner
+EXTRA_DIST += hammer.test/hammer.exp
+EXTRA_DIST += spanner.test/spanner.exp
+END
+
+cat > hammer << 'END'
+#! /bin/sh
+echo "Everything looks like a ${NAIL-nail} to me!"
+END
+
+NAIL=nail; export NAIL
+
+cat > spanner << 'END'
+#! /bin/sh
+echo "I'm a right spanner!"
+END
+chmod +x hammer spanner
+
+mkdir hammer.test spanner.test
+
+cat > hammer.test/hammer.exp << 'END'
+set test test_hammer
+spawn $HAMMER
+expect {
+    "Everything looks like a nail to me!" { pass "$test" }
+    default { fail "$test" }
+}
+END
+
+cat > spanner.test/spanner.exp << 'END'
+set test test_spanner
+spawn $SPANNER
+expect {
+    "I'm a right spanner!" { pass "$test" }
+    default { fail "$test" }
+}
+END
+
+## User-defined extra tests.
+
+cat >> Makefile.am <<'END'
+check-local:
+       case $$CHECKLOCAL_EXIT_STATUS in \
+         0) echo 'check-local succeded :-)';; \
+         *) echo 'check-local failed :-(';; \
+       esac >local.log
+       exit $$CHECKLOCAL_EXIT_STATUS
+CLEANFILES += local.log
+END
+CHECKLOCAL_EXIT_STATUS=0; export CHECKLOCAL_EXIT_STATUS
+
+## Go with the testcase execution.
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+for vpath in : false; do
+
+  if $vpath; then
+    srcdir=..
+    mkdir build_
+    cd build_
+  else
+    srcdir=.
+  fi
+
+  $srcdir/configure
+
+  $MAKE check
+  test -f hammer.log
+  test -f hammer.sum
+  test -f spanner.log
+  test -f spanner.sum
+  # This checks will be run only by the autogenerated `check12-p.test'.
+  if test x"$parallel_tests" = x"yes"; then
+    cat test-suite.log
+    grep '^PASS: a\.test (exit: 0)' a.log
+    grep '^PASS: b\.test (exit: 0)' b.log
+  else
+    :
+  fi
+  grep 'check-local succeded :-)' local.log
+
+  cp -f config.status config-status.sav
+
+  $MAKE distclean
+  test ! -r hammer.log
+  test ! -r hammer.sum
+  test ! -r spanner.log
+  test ! -r spanner.sum
+  test ! -r test-suite.log
+  test ! -r a.log
+  test ! -r b.log
+  test ! -r local.log
+
+  mv -f config-status.sav config.status
+  ./config.status
+
+  NAIL=screw $MAKE check && Exit 1
+  test -f hammer.log
+  test -f hammer.sum
+  test -f spanner.log
+  test -f spanner.sum
+  grep 'FAIL: test_hammer' hammer.sum
+  grep 'FAIL:' spanner.sum && Exit 1
+
+  B_EXIT_STATUS=1 $MAKE check && Exit 1
+  # This checks will be run only by the autogenerated `check12-p.test'.
+  if test x"$parallel_tests" = x"yes"; then
+    cat test-suite.log
+    grep '^PASS: a\.test (exit: 0)' a.log
+    grep '^FAIL: b\.test (exit: 1)' b.log
+    grep '^FAIL: b\.test (exit: 1)' test-suite.log
+  else :; fi
+
+  CHECKLOCAL_EXIT_STATUS=1 $MAKE check && Exit 1
+  grep 'check-local failed :-(' local.log
+
+  # Do not trust the exit status of `make -k'.
+  NAIL=screw B_EXIT_STATUS=23 CHECKLOCAL_EXIT_STATUS=1 $MAKE -k check || :
+  test -f hammer.log
+  test -f hammer.sum
+  test -f spanner.log
+  test -f spanner.sum
+  grep 'FAIL: test_hammer' hammer.sum
+  grep 'FAIL:' spanner.sum && Exit 1
+  # This checks will be run only by the autogenerated `check12-p.test'.
+  if test x"$parallel_tests" = x"yes"; then
+    cat test-suite.log
+    grep '^PASS: a\.test (exit: 0)' a.log
+    grep '^FAIL: b\.test (exit: 23)' b.log
+    grep '^FAIL: b\.test (exit: 23)' test-suite.log
+  else :; fi
+  grep 'check-local failed :-(' local.log
+
+  $MAKE distcheck
+  $MAKE distclean
+  cd $srcdir
+
+done
+
+:
diff --git a/tests/dejagnu.test b/tests/dejagnu.test
index a273770..b3e58e7 100755
--- a/tests/dejagnu.test
+++ b/tests/dejagnu.test
@@ -14,7 +14,8 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-# Test to make sure dejagnu tests and automake-style tests can coexist.
+# Superficial test to check that dejagnu tests and automake-style
+# tests can coexist.  See also related deeper test `check12.test'.
 
 . ./defs || Exit 1
 
@@ -25,10 +26,10 @@ AUTOMAKE_OPTIONS = dejagnu
 TESTS = frob.test
 END
 
-: > frob.test
-
 $ACLOCAL
 $AUTOMAKE
 
 grep '^check-TESTS' Makefile.in
 grep '^check-DEJAGNU' Makefile.in
+
+:
diff --git a/tests/dejagnu2.test b/tests/dejagnu2.test
index 531f744..121f6d8 100755
--- a/tests/dejagnu2.test
+++ b/tests/dejagnu2.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2001, 2002, 2003  Free Software Foundation, Inc.
+# Copyright (C) 2001, 2002, 2003, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,26 +20,29 @@
 
 set -e
 
-echo 'AC_OUTPUT' >> configure.in
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
 
 site.exp:
-       @echo foo
+       echo :GREP:ME: > $@
 END
 
 $ACLOCAL
 $AUTOCONF
 $AUTOMAKE -Wno-override
 
-grep site.exp Makefile.in
-test `grep '^site\.exp:' Makefile.in | wc -l` -eq 1
+grep 'site\.exp' Makefile.in
+test `grep -c '^site\.exp:' Makefile.in` -eq 1
 
 ./configure
-$MAKE site.exp >stdout
-cat stdout
-grep foo stdout
+$MAKE site.exp
+grep ':GREP:ME:' site.exp
 
 AUTOMAKE_fails
-grep 'Makefile.am:3:.*site.exp' stderr
+grep '^Makefile\.am:3:.*site\.exp' stderr
+
+:
diff --git a/tests/dejagnu3.test b/tests/dejagnu3.test
index d812ac4..2c5d375 100755
--- a/tests/dejagnu3.test
+++ b/tests/dejagnu3.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -28,15 +28,14 @@ END
 
 chmod +x hammer
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
-
 DEJATOOL = hammer
-
 AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer
-
 EXTRA_DIST = hammer hammer.test/hammer.exp
 END
 
@@ -65,5 +64,9 @@ $MAKE distcheck
 
 # Ensure that the envvar RUNTESTFLAGS is used.
 # Report from Mark Mitchell.
-RUNTESTFLAGS=--unknown-option $MAKE check && Exit 1
-Exit 0
+RUNTESTFLAGS=--unknown-runtest-option $MAKE check >output 2>&1 \
+  && { cat output; Exit 1; }
+cat output
+$FGREP 'unknown-runtest-option' output
+
+:
diff --git a/tests/dejagnu4.test b/tests/dejagnu4.test
index 5fdda10..fa64a31 100755
--- a/tests/dejagnu4.test
+++ b/tests/dejagnu4.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003, 2006  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2006, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -26,17 +26,17 @@ cat > hammer << 'END'
 #! /bin/sh
 echo "Everything looks like a nail to me!"
 END
-
 chmod +x hammer
 
 cat > spanner << 'END'
 #! /bin/sh
 echo "I'm a right spanner!"
 END
-
 chmod +x spanner
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
@@ -49,8 +49,7 @@ EXTRA_DIST  = hammer  hammer.test/hammer.exp
 EXTRA_DIST += spanner spanner.test/spanner.exp
 END
 
-mkdir hammer.test
-mkdir spanner.test
+mkdir hammer.test spanner.test
 
 cat > hammer.test/hammer.exp << 'END'
 set test test
@@ -86,8 +85,14 @@ $MAKE distcheck
 
 # Test for PR 488.
 sed 's/E\(verything\)/Not e\1/' hammer > thammer
-mv thammer hammer
+mv -f thammer hammer
 chmod +x hammer
 
+rm -f hammer.log hammer.sum spanner.log spanner.sum
 $MAKE check && Exit 1
+test -f hammer.log
+test -f hammer.sum
+test -f spanner.log
+test -f spanner.sum
+
 :
diff --git a/tests/dejagnu5.test b/tests/dejagnu5.test
index 0095958..0237a08 100755
--- a/tests/dejagnu5.test
+++ b/tests/dejagnu5.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -22,30 +22,27 @@ required=runtest
 
 set -e
 
-# Get the package name from configure.in.
-package=`$FGREP AC_INIT configure.in | sed -e 's:^[^[]*\[\([^]]*\)\].*$:\1:'`
+package=$me
 
 cat > $package << 'END'
 #! /bin/sh
 echo "Ah, we have been expecting you, Mr. Blond."
 END
-
 chmod +x $package
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << END
 AUTOMAKE_OPTIONS = dejagnu
-
 EXTRA_DIST = $package $package.test/$package.exp
+AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package
 END
 
-echo "AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package" >> Makefile.am
-
 mkdir $package.test
-
 cat > $package.test/$package.exp << 'END'
-set test "test"
+set test "a_dejagnu_test"
 spawn $PACKAGE
 expect {
     "Ah, we have been expecting you, Mr. Blond." { pass "$test" }
@@ -64,3 +61,5 @@ test -f $package.log
 test -f $package.sum
 
 $MAKE distcheck
+
+:
diff --git a/tests/dejagnu6.test b/tests/dejagnu6.test
index 25a1f1f..662ca28 100755
--- a/tests/dejagnu6.test
+++ b/tests/dejagnu6.test
@@ -1,5 +1,5 @@
 #! /bin/sh
-# Copyright (C) 2003  Free Software Foundation, Inc.
+# Copyright (C) 2003, 2010 Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -25,23 +25,21 @@ cat > faildeja << 'END'
 #! /bin/sh
 echo whatever
 END
-
 chmod +x faildeja
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
-
 DEJATOOL = faildeja
-
 AM_RUNTESTFLAGS = FAILDEJA=$(srcdir)/faildeja
 END
 
 mkdir faildeja.test
-
 cat > faildeja.test/faildeja.exp << 'END'
-set test test
+set test failing_deja_test
 spawn $FAILDEJA
 expect {
     default { fail "$test" }
@@ -57,4 +55,6 @@ $AUTOMAKE --add-missing
 $MAKE check && Exit 1
 test -f faildeja.log
 test -f faildeja.sum
-$FGREP 'FAIL: test' faildeja.sum
+$FGREP 'FAIL: failing_deja_test' faildeja.sum
+
+:
diff --git a/tests/dejagnu7.test b/tests/dejagnu7.test
index 552c2ca..0be4d1f 100755
--- a/tests/dejagnu7.test
+++ b/tests/dejagnu7.test
@@ -29,21 +29,19 @@ cat > failtcl << 'END'
 #! /bin/sh
 echo whatever
 END
-
 chmod +x failtcl
 
-echo AC_OUTPUT >> configure.in
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
 
 cat > Makefile.am << 'END'
 AUTOMAKE_OPTIONS = dejagnu
-
 DEJATOOL = failtcl
-
 AM_RUNTESTFLAGS = --status FAILTCL=$(srcdir)/failtcl
 END
 
 mkdir failtcl.test
-
 cat > failtcl.test/failtcl.exp << 'END'
 set test test
 spawn $FAILTCL
@@ -62,3 +60,5 @@ $MAKE check && Exit 1
 test -f failtcl.log
 test -f failtcl.sum
 $FGREP 'missing close-brace' failtcl.sum
+
+:
-- 
1.7.1