Re: autotest annoyances

[Eric Blake]

> 
> Obviously, this would also need more tests in autotest.at.
> And I think the first test can be tightened to test only
> missing ./run, instead of getting mixed up with invalid ./run,
>  even if my proposal [for fixing ./testsuite return codes] is
>  too invasive before the alpha release.

> So my resubmittal will have to touch up that documentation as well.
> I think that the best semantics is that running the debugging script
> regenerates the entire directory, including a fresh rebuild of ./run,
> and with a new timestamp for testsuite.log; but that since the
> re-run uses the same command line options, the same results
> should happen.
> 

Here is an updated version of my previous patch.  I have avoided
using environment variables when testing that ./run exists in the
first test, so that we don't get a pass for the wrong reason.  And
-d DOES inhibit logging, but only at the top-level (in order to
preserve the full log).  At the testgroup level, since the entire
testgroup directory is blown away, the testgroup log is regenerated
even when -d is specified.

2006-04-08  Eric Blake  <address@hidden>

        * tests/autotest.at (AT_CHECK_AT): Add new argument, to allow
        top-level tests after micro-suite has been run.   Used in...
        (Debugging a successful test, Debugging script and environment),
        (Debugging a failed test): ...these new tests.  The first of these
        is fixed by...
        * lib/autotest/general.m4 (_AT_CREATE_DEBUGGING_SCRIPT): New
        macro, split out from...
        (AT_INIT): ...here, so that using -d also generates a run script.
        Document that -d inhibits top-level logging.
        * doc/autoconf.texi (testsuite Invocation): Document that -d only
        inhibits top-level logging; debug scripts are created.

-- 
Eric Blake

Index: tests/autotest.at
===================================================================
RCS file: /sources/autoconf/autoconf/tests/autotest.at,v
retrieving revision 1.11
diff -u -p -r1.11 autotest.at
--- tests/autotest.at   5 Apr 2006 16:04:22 -0000       1.11
+++ tests/autotest.at   8 Apr 2006 18:38:37 -0000
@@ -20,12 +20,13 @@ AT_BANNER([Autotest.])
 # 02110-1301, USA.
 
 # AT_CHECK_AT(TITLE, SUITE-CODE, [XFAIL-CONDITION], [STATUS = 0],
-#             [STDOUT := ignore], STDERR)
+#             [STDOUT := ignore], STDERR, [POST-TEST-CODE])
 # ---------------------------------------------------------------
 # Create a new test named TITLE that runs a minimal Autotest test suite,
 # SUITE-CODE.  Call AT_XFAIL_IF with XFAIL-CONDITION.  STATUS and STDERR pass
 # directly to the AT_CHECK that call the minimal test suite.  STDOUT is not
-# used, but it is reserved for future use.
+# used, but it is reserved for future use.  Run POST-TEST-CODE
+# at the top level after the micro-suite has been run.
 m4_define([AT_CHECK_AT],
 [
 AT_SETUP([$1])
@@ -49,8 +50,9 @@ AT_DATA([mysuite.at], [$2])
 AT_CHECK_AUTOM4TE([--language=autotest -o micro-suite mysuite.at])
 AT_CHECK([$CONFIG_SHELL ./micro-suite],       m4_default([$4], 0), [ignore], 
[$6])
 AT_CHECK([$CONFIG_SHELL ./micro-suite -v -x], m4_default([$4], 0), [ignore], 
[$6])
+$7
 AT_CLEANUP
-])
+])# AT_CHECK_AT
 
 # AT_CHECK_AT_TEST(TITLE, SUITE-SNIPPET, ...)
 # -----------------------------------------------------------------------
@@ -251,6 +253,51 @@ AT_CHECK_AT_TITLE_CHAR([Double-quote], [
 AT_CHECK_AT_TITLE_CHAR([Backslash],    [\])
 
 
+## ----------------- ##
+## Debugging a test. ##
+## ----------------- ##
+
+AT_CHECK_AT_TEST([Debugging a successful test],
+  [AT_CHECK([:])], [], [], [], [ignore],
+[# Without options, when all tests pass, no test directory should exist.
+AT_CHECK([test -d micro-suite.dir/1 && exit 42
+          ./micro-suite -d 1], [], [ignore], [ignore])
+# Running with -d should leave a reproducible test group.
+# Also, running the test script from the test group locks the
+# directory from removal on some platforms; the script should still be
+# able to run even if rmdir fails.
+AT_CHECK([(cd micro-suite.dir/1 && ./run)], [], [ignore], [ignore])
+# Running a debugging script implies -d.
+AT_CHECK([(cd micro-suite.dir/1 && ./run)], [], [ignore], [ignore])
+])
+
+AT_CHECK_AT_TEST([Debugging script and environment],
+  [AT_CHECK([test "$MY_VAR" = pass || exit 42])],
+  [], [1], [], [ignore], [
+# Changing environment outside of debugging script is not preserved.
+AT_CHECK([(cd micro-suite.dir/1 && MY_VAR=pass ./run)],
+         [0], [ignore], [ignore])
+AT_CHECK([(cd micro-suite.dir/1 && ./run)],
+         [1], [ignore], [ignore])
+# Changing environment as argument to debugging script is preserved.
+AT_CHECK([(cd micro-suite.dir/1; ./run MY_VAR=pass)],
+         [0], [ignore], [ignore])
+AT_CHECK([(cd micro-suite.dir/1; ./run)],
+         [0], [ignore], [ignore])
+])
+
+# The run script is currently invalid when shell metacharacters are passed
+# in via an environment option.
+AT_CHECK_AT_TEST([Debugging a failed test],
+  [AT_CHECK([test "$MY_VAR" = "one space" || exit 42])],
+  [:], [1], [], [ignore], [
+AT_CHECK([(cd micro-suite.dir/1 && ./run MY_VAR='two  spaces')],
+         [1], [ignore], [ignore])
+AT_CHECK([(cd micro-suite.dir/1 && ./run MY_VAR='one space')],
+         [0], [ignore], [ignore])
+])
+
+
 ## --------- ##
 ## Keywords. ##
 ## --------- ##
Index: lib/autotest/general.m4
===================================================================
RCS file: /sources/autoconf/autoconf/lib/autotest/general.m4,v
retrieving revision 1.206
diff -u -p -r1.206 general.m4
--- lib/autotest/general.m4     7 Apr 2006 18:31:09 -0000       1.206
+++ lib/autotest/general.m4     8 Apr 2006 18:38:38 -0000
@@ -162,6 +162,22 @@ m4_define([_AT_NORMALIZE_TEST_GROUP_NUMB
   done
 ])
 
+# _AT_CREATE_DEBUGGING_SCRIPT
+# ---------------------------
+# Create the debugging script $at_group_dir/run which will reproduce the
+# current test group.
+m4_define([_AT_CREATE_DEBUGGING_SCRIPT],
+[        {
+           echo "#! /bin/sh"
+           echo 'test "${ZSH_VERSION+set}" = set && alias -g 
'\''${1+"address@hidden"}'\''='\''"address@hidden"'\'''
+           echo "cd '$at_dir'"
+           echo 'exec ${CONFIG_SHELL-'"$SHELL"'}' "$[0]" \
+                '-v -d' "$at_debug_args" "$at_group" '${1+"address@hidden"}'
+           echo 'exit 1'
+         } >$at_group_dir/run
+         chmod +x $at_group_dir/run
+])# _AT_CREATE_DEBUGGING_SCRIPT
+
 
 # AT_INIT([TESTSUITE-NAME])
 # -------------------------
@@ -481,7 +497,7 @@ Execution tuning:
   -e, --errexit  abort as soon as a test fails; implies --debug
   -v, --verbose  force more detailed output
                 default for debugging scripts
-  -d, --debug    inhibit clean up and debug script creation
+  -d, --debug    inhibit clean up and top-level logging
                 default for debugging scripts
   -x, --trace    enable tests shell tracing
 _ATEOF
@@ -793,11 +809,12 @@ _ATEOF
          echo "$at_log_msg" >&AS_MESSAGE_LOG_FD
 
          # Cleanup the group directory, unless the user wants the files.
-         $at_debug_p ||
-           if test -d $at_group_dir; then
-             find $at_group_dir -type d ! -perm -700 -exec chmod u+rwx \{\} \;
-             rm -fr $at_group_dir
-           fi
+         if $at_debug_p ; then
+           _AT_CREATE_DEBUGGING_SCRIPT
+         elif test -d $at_group_dir; then
+           find $at_group_dir -type d ! -perm -700 -exec chmod u+rwx \{\} \;
+           rm -fr $at_group_dir
+         fi
          ;;
        *)
          # Upon failure, include the log into the testsuite's global
@@ -807,15 +824,7 @@ _ATEOF
 
          # Upon failure, keep the group directory for autopsy, and
          # create the debugging script.
-         {
-           echo "#! /bin/sh"
-           echo 'test "${ZSH_VERSION+set}" = set && alias -g 
'\''${1+"address@hidden"}'\''='\''"address@hidden"'\'''
-           echo "cd '$at_dir'"
-           echo 'exec ${CONFIG_SHELL-'"$SHELL"'}' "$[0]" \
-                '-v -d' "$at_debug_args" "$at_group" '${1+"address@hidden"}'
-           echo 'exit 1'
-         } >$at_group_dir/run
-         chmod +x $at_group_dir/run
+         _AT_CREATE_DEBUGGING_SCRIPT
          $at_errexit && break
          ;;
       esac
Index: doc/autoconf.texi
===================================================================
RCS file: /sources/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.988
diff -u -p -r1.988 autoconf.texi
--- doc/autoconf.texi   7 Apr 2006 18:00:55 -0000       1.988
+++ doc/autoconf.texi   8 Apr 2006 18:38:42 -0000
@@ -16913,8 +16913,8 @@ containing @samp{FUNC}.
 @item --errexit
 @itemx -e
 If any test fails, immediately abort testing.  It implies
address@hidden: post test group clean up, debugging script generation,
-and logging are inhibited.  This option is meant for the full test
address@hidden: post test group clean up, and top-level logging
+are inhibited.  This option is meant for the full test
 suite, it is not really useful for generated debugging scripts.
 
 @item --verbose
@@ -16926,7 +16926,8 @@ is the default for debugging scripts.
 @itemx -d
 Do not remove the files after a test group was performed ---but they are
 still removed @emph{before}, therefore using this option is sane when
-running several test groups.  Do not create debugging scripts.  Do not
+running several test groups.  Create debugging scripts.  Do not
+overwrite the top-level
 log (in order to preserve supposedly existing full log file).  This is
 the default for debugging scripts, but it can also be useful to debug
 the testsuite itself.