48-fyi-test-logs.patch

[Akim Demaille]

Index: ChangeLog
from  Akim Demaille  <address@hidden>

        * doc/autoconf.texi (Using an Autotest Test Suite): New.
        (testsuite Scripts): Be one of its subsection.
        (Autotest Logs): New.

Index: doc/autoconf.texi
--- doc/autoconf.texi Sat, 24 Nov 2001 18:33:18 +0100 akim
+++ doc/autoconf.texi Sat, 24 Nov 2001 18:34:17 +0100 akim
@@ -168,7 +168,8 @@ @node Top
 * Copying This Manual::         How to make copies of this manual
 * Indices::                     Indices of symbols, concepts, etc.

address@hidden --- The Detailed Node Listing ---
address@hidden
+ --- The Detailed Node Listing ---

 The GNU build system

@@ -408,11 +409,16 @@ @node Top

 Generating Test Suites with Autotest

-* testsuite Scripts::           The concepts of Autotest
+* Using an Autotest Test Suite::  Autotest and the user
 * Writing testsuite.at::        Autotest macros
 * testsuite Invocation::        Running @command{testsuite} scripts
 * Making testsuite Scripts::    Using autom4te to create @command{testsuite}

+Using an Autotest Test Suite
+
+* testsuite Scripts::           The concepts of Autotest
+* Autotest Logs::               Their contents
+
 Questions About Autoconf

 * Distributing::                Distributing @command{configure} scripts
@@ -11431,14 +11437,22 @@ problem: although it aims at perfectly p
 tool testing, which is probably its main limitation.

 @menu
-* testsuite Scripts::           The concepts of Autotest
+* Using an Autotest Test Suite::  Autotest and the user
 * Writing testsuite.at::        Autotest macros
 * testsuite Invocation::        Running @command{testsuite} scripts
 * Making testsuite Scripts::    Using autom4te to create @command{testsuite}
 @end menu

address@hidden Using an Autotest Test Suite
address@hidden Using an Autotest Test Suite
+
address@hidden
+* testsuite Scripts::           The concepts of Autotest
+* Autotest Logs::               Their contents
address@hidden menu
+
 @node testsuite Scripts
address@hidden @command{testsuite} Scripts
address@hidden @command{testsuite} Scripts

 @cindex @command{testsuite}

@@ -11537,6 +11551,65 @@ subfile-n.at ->'                       >
              /                  \
 [atlocal] ->'                    `--> address@hidden
 @end example
+
+
address@hidden Autotest Logs
address@hidden Autotest Logs
+
+When run, the test suite creates a log file named after itself, e.g., a
+test suite named @command{testsuite} creates @file{testsuite.log}.  It
+contains a lot of information, usually more than maintainers actually
+need, but therefore most of the time it contains all that is needed:
+
address@hidden @asis
address@hidden command line arguments
address@hidden akim s/to consist in/to consist of/
+A very bad Unix habit which is unfortunately wide spread consists of
+setting environment variables before the command, such as in
address@hidden ./testsuite}.  This results in the test suite
+not knowing this change, hence (i) it can't report it to you, and (ii)
+it cannot preserve the value of @code{CC} for subsequent address@hidden
address@hidden
+When a failure occurs, the test suite is rerun, verbosely, and the user
+is asked to ``play'' with this failure to provide better information.
+It is important to keep the same environment between the first run, and
+bug-tracking runs.
address@hidden
+}.  Autoconf faced exactly the same problem, and solved it by asking
+users to pass the variable definitions as command line arguments.
+Autotest requires this rule too, but has no means to enforce it; the log
+then contains a trace of the variables the user changed.
+
address@hidden @file{ChangeLog} excerpts
+The topmost lines of all the @file{ChangeLog}s found in the source
+hierarchy.  This is especially useful when bugs are reported against
+development versions of the package, since the version string does not
+provide sufficient information to know the exact state of the sources
+the user compiled.  Of course this relies on the use of a
address@hidden
+
address@hidden build machine
+Running a test suite in a cross-compile environment is not an easy task,
+since it would mean having the test suite run on a machine @var{build},
+while running programs on a machine @var{host}.  It is much simpler to
+run both the test suite and the programs on @var{host}, but then, from
+the point of view of the test suite, there remains a single environment,
address@hidden = @var{build}.  The log contains relevant information on the
+state of the build machine, including some important environment
+variables.
address@hidden FIXME: How about having an M4sh macro to say `hey, log the value
address@hidden of `...'?  This would help both Autoconf and Autotest.
+
address@hidden tested programs
+The absolute path and answers to @option{--version} of the tested
+programs (see @ref{Writing testsuite.at}, @code{AT_TESTED}).
+
address@hidden configuration log
+The contents of @file{config.log}, as created by @command{configure},
+are appended.  It contains the configuration flags and a detailed report
+on the configuration itself.
address@hidden table
+

 @node Writing testsuite.at
 @section Writing @file{testsuite.at}