[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: program transform and tests
|
From: |
Ralf Wildenhues |
|
Subject: |
Re: program transform and tests |
|
Date: |
Tue, 9 May 2006 18:59:12 +0200 |
|
User-agent: |
Mutt/1.5.11 |
Hi Paul, everyone,
* Paul Eggert wrote on Mon, May 01, 2006 at 07:13:27AM CEST:
> Ralf Wildenhues <address@hidden> writes:
>
> > Is it worthwhile to have
> > make installcheck
> >
> > do the right thing in the presence of
> > configure --program-transform-name=...
> >
> > post-2.60, or even before that? Fixing that touches quite a few tests,
> > and makes `testsuite -x' output somewhat less readable by introducing
> > some shell variables.
>
> Do people actually use --program-transform-name for autoconf?
> If so, it's probably worth fixing now, assuming the fixes are
> mechanical and are not too dangerous.
Summary: I've looked at this, and have a half-baked patch. It's doable,
but a bit ugly, and I haven't found any non-testsuite changes necessary.
IMHO not useful to have before the release, since all it's about to
introduce are potential bugs.
But for the curious, here's the patch and associated description:
The first bugfix is that
./configure && make && make install && make installcheck
does not build `tests/autom4te', but that is necessary for creating
`tests/testsuite'. I think that is wrong, and the testsuite should
be created by the same procedure by which the frozen files are created.
Then, we don't want to interfere with the values of $(AUTOCONF) etc.
that stem from Automake. And it'd be nice to be able to override
the automake and aclocal versions for testing. And we do not want to
set AUTOCONF anyway in order to test the normal code paths which don't
have these variables set.
So one possibility is to use other variables for all of the tools; I
chose lower-case ones in the patch below (but that is certainly up to
debate). However, since automake and aclocal may invoke autom4te, and
since autoreconf may invoke them in turn, AUTOMAKE, ACLOCAL, and
AUTOM4TE have to be set, in order to maintain sanity.
The patch further exposes a weakness in AT_TESTED which I reported
almost a year ago: it's not a good idea to only allow M4 literals as
arguments, and it's really not a good idea to fail the testsuite hard
if they don't work. Since nobody bothered to fix this until now, I
think it's safe to postpone until post-2.60 too, though.
Also note I did not bother to make the "Syntax of the shell scripts"
and "Syntax of the Perl scripts" test the installed files.
Finally, some notes as to how I made sure this patch was complete:
configure with some --program-transform, install, prepend PATH with a
directory which has executables for all Autoconf programs that fail,
e.g.,
#! /bin/sh
exit 1
so that no system-installed programs interfere (no need to have $bindir
in the PATH); make sure $top_builddir/tests/ does not contain the
wrappers that `make check' will cause to be present (i.e., do not run
`make check'). Then, run `make installcheck'.
(For normal testing, this procedure isn't necessary of course.)
After all this, I ran
make installcheck automake=automake-1.5 aclocal=aclocal-1.5
make installcheck automake=automake-1.7 aclocal=aclocal-1.7
on a GNU/Linux system, and found only one failure with both of them:
| 94: torture.at:972 Unusual Automake input files
which fails because the Automake macros in that test use newer syntax;
the test should not be adjusted because it aims to parallel a test in
Automake's test suite.
Cheers,
Ralf
* tests/Makefile.am (AUTOTEST): Use `$(MY_AUTOM4TE)' instead of
`./autom4te' to create `./testsuite', since the `all' target
will ensure its presence, but `installcheck' should not create
the uninstalled wrappers.
(BUILDCHECK_ENVIRONMENT, INSTALLCHECK_ENVIRONMENT): New macros
to specify..
(autom4te, autoconf, autoheader, autoupdate, autoreconf)
(autoscan, ifnames, aclocal, automake, ACLOCAL, AUTOMAKE)
(AUTOM4TE): ..these new variables to be passed to the testsuite
(check-local, installcheck-local): ..here.
* tests/autoscan.at, tests/local.at, tests/m4sh.at: Use them.
* tests/tools.at, tests/torture.at: Likewise.
* tests/local.at: Add a note that the usage of AT_TESTED is
unsafe.
(AT_CHECK_ACLOCAL): New macro.
* tests/tools.at: Use it.
Index: tests/Makefile.am
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/Makefile.am,v
retrieving revision 1.100
diff -u -r1.100 Makefile.am
--- tests/Makefile.am 9 Apr 2006 11:36:08 -0000 1.100
+++ tests/Makefile.am 9 May 2006 16:31:40 -0000
@@ -109,7 +109,7 @@
# Run the non installed autom4te.
# Don't use AUTOM4TE since `make alpha' makes it unavailable although
# we are allowed to use it (since we ship it).
-AUTOTEST = ./autom4te --language=autotest
+AUTOTEST = $(MY_AUTOM4TE) --language=autotest
$(TESTSUITE): $(srcdir)/package.m4 \
local.at \
$(TESTSUITE_AT) \
@@ -126,12 +126,53 @@
rm -f *.tmp
rm -f -r autom4te.cache
+
+# `installcheck' should use the transformed installed programs,
+# and, e.g., autoreconf should find the transformed autoconf without
+# $AUTOCONF being set. However, aclocal may invoke autom4te, and we
+# cannot control aclocal, thus we have to let it know about this.
+# Also, for consistency, we set $automake and $aclocal, but here we
+# have to respect the uppercase variables, if set. We do not use
+# $ACLOCAL because that may interfere with how this package was
+# configured: that should be independent of the Automake package
+# we test with. We initialize $ACLOCAL and $AUTOMAKE inside the
+# test suite.
+BUILDCHECK_ENVIRONMENT = \
+ autom4te=autom4te \
+ autoconf=autoconf \
+ autoheader=autoheader \
+ autoupdate=autoupdate \
+ autoreconf=autoreconf \
+ autoscan=autoscan \
+ ifnames=ifnames \
+ aclocal="$${aclocal-aclocal}" \
+ automake="$${automake-automake}" \
+ ACLOCAL="$${aclocal-aclocal}" \
+ AUTOMAKE="$${automake-automake}" \
+ AUTOM4TE=autom4te
+
+INSTALLCHECK_ENVIRONMENT = \
+ autom4te=`echo autom4te | sed '$(transform)'` \
+ autoconf=`echo autoconf | sed '$(transform)'` \
+ autoheader=`echo autoheader | sed '$(transform)'` \
+ autoupdate=`echo autoupdate | sed '$(transform)'` \
+ autoreconf=`echo autoreconf | sed '$(transform)'` \
+ autoscan=`echo autoscan | sed '$(transform)'` \
+ ifnames=`echo ifnames | sed '$(transform)'` \
+ aclocal="$${aclocal-aclocal}" \
+ automake="$${automake-automake}" \
+ ACLOCAL="$${aclocal-aclocal}" \
+ AUTOMAKE="$${automake-automake}" \
+ AUTOM4TE=`echo autom4te | sed '$(transform)'`
+
check-local: atconfig atlocal $(TESTSUITE)
- $(SHELL) $(TESTSUITE) $(TESTSUITEFLAGS)
+ $(SHELL) $(TESTSUITE) $(TESTSUITEFLAGS) \
+ $(BUILDCHECK_ENVIRONMENT)
# Run the test suite on the *installed* tree.
installcheck-local: atconfig atlocal $(TESTSUITE)
- $(SHELL) $(TESTSUITE) AUTOTEST_PATH="$(bindir)" $(TESTSUITEFLAGS)
+ $(SHELL) $(TESTSUITE) AUTOTEST_PATH="$(bindir)" $(TESTSUITEFLAGS) \
+ $(INSTALLCHECK_ENVIRONMENT)
Index: tests/autoscan.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/autoscan.at,v
retrieving revision 1.3
diff -u -r1.3 autoscan.at
--- tests/autoscan.at 3 Apr 2006 15:58:20 -0000 1.3
+++ tests/autoscan.at 9 May 2006 16:31:40 -0000
@@ -43,7 +43,7 @@
AC_CONFIG_FILES(Makefile)
AC_OUTPUT
]])
-AT_CHECK([autoscan])
+AT_CHECK([$autoscan])
AT_CHECK([grep subpkg/Makefile configure.scan], [1], [], [ignore])
AT_CLEANUP
Index: tests/local.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/local.at,v
retrieving revision 1.30
diff -u -r1.30 local.at
--- tests/local.at 28 Apr 2006 04:17:51 -0000 1.30
+++ tests/local.at 9 May 2006 16:31:40 -0000
@@ -25,6 +25,7 @@
m4_pattern_allow([^AS_EXIT$])
m4_pattern_allow([^m4_(define|shift)$])
+dnl FIXME: this does not take program-transform-name into account
AT_TESTED([autom4te autoconf autoheader autoupdate autoreconf ifnames])
@@ -89,12 +90,12 @@
# autom4te: [^ ]m4
# to autom4te: m4
m4_define([AT_CHECK_AUTOM4TE],
-[AT_CHECK([autom4te $1], [$2], [$3], m4_ifval([$4], [stderr]))
+[AT_CHECK([$autom4te $1], [$2], [$3], m4_ifval([$4], [stderr]))
m4_ifval([$4],
[AT_CHECK([[sed -e 's/^\([^:]*\): *\([0-9][0-9]*\): *[^:]*m4: /m4: \1: \2: /' \
-e 's/^[^:]*m4: *\([^:]*\): *\([0-9][0-9]*\): /m4: \1: \2: /' \
- -e 's/^autom4te: [^ ]*m4 /autom4te: m4 /' \
- -e 's/^autom4te: [^ ]*m4.exe /autom4te: m4 /' \
+ -e 's/^'"$autom4te"': [^ ]*m4 /autom4te: m4 /' \
+ -e 's/^'"$autom4te"': [^ ]*m4.exe /autom4te: m4 /' \
-e 's/ (E[A-Z]*)$//' \
stderr]], [0],[$4])])
])
@@ -203,7 +204,7 @@
# We always use "--force", to prevent problems with timestamps if the testsuite
# were running too fast.
m4_define([AT_CHECK_AUTOCONF],
-[AT_CHECK([autoconf --force $1], [$2], [$3], [$4])
+[AT_CHECK([$autoconf --force $1], [$2], [$3], [$4])
if test -s configure; then
AT_CHECK_SHELL_SYNTAX(configure)
fi
@@ -213,7 +214,7 @@
# AT_CHECK_AUTOHEADER(ARGS, [EXIT-STATUS = 0], STDOUT, STDERR)
# ------------------------------------------------------------
m4_define([AT_CHECK_AUTOHEADER],
-[AT_CHECK([autoheader $1], [$2], [$3], [$4])
+[AT_CHECK([$autoheader $1], [$2], [$3], [$4])
])
@@ -372,7 +373,7 @@
# AT_CHECK_AUTOUPDATE
# -------------------
m4_define([AT_CHECK_AUTOUPDATE],
-[AT_CHECK([autoupdate $1], [$2], [$3], [$4])
+[AT_CHECK([$autoupdate $1], [$2], [$3], [$4])
])
@@ -479,6 +480,13 @@
])# AT_CHECK_AU_MACRO
+# AT_CHECK_ACLOCAL(ARGS, [EXIT-STATUS = 0], STDOUT, STDERR)
+# ---------------------------------------------------------
+m4_define([AT_CHECK_ACLOCAL],
+[AT_CHECK([$aclocal $1], [$2], [$3], [$4])
+])
+
+
## ----------------------- ##
## Launch the test suite. ##
Index: tests/m4sh.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/m4sh.at,v
retrieving revision 1.47
diff -u -r1.47 m4sh.at
--- tests/m4sh.at 24 Apr 2006 19:36:02 -0000 1.47
+++ tests/m4sh.at 9 May 2006 16:31:40 -0000
@@ -62,7 +62,7 @@
]])
# If occurrences of $LINENO or __oline__ were wanted, create them.
sed 's/__LINENO__/$''LINENO/g;s/__OLINE__/__''oline__/g' $1.tas >$1.as
-AT_CHECK([autom4te -l m4sh $1.as -o $1])
+AT_CHECK([$autom4te -l m4sh $1.as -o $1])
])# AT_DATA_LINENO
# `_oline_', once processed and ran, produces our reference.
Index: tests/tools.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/tools.at,v
retrieving revision 1.83
diff -u -r1.83 tools.at
--- tests/tools.at 19 Apr 2006 04:13:48 -0000 1.83
+++ tests/tools.at 9 May 2006 16:31:40 -0000
@@ -52,8 +52,10 @@
# running `sh PROG'.
AT_CHECK_SHELL_SYNTAX([$abs_top_builddir/bin/autoconf])
-AT_CHECK_SHELL_SYNTAX([$abs_top_builddir/tests/autoconf])
-AT_CHECK_SHELL_SYNTAX([$abs_top_builddir/tests/testsuite])
+AS_IF([test -f $abs_top_builddir/tests/autoconf],
+ [AT_CHECK_SHELL_SYNTAX([$abs_top_builddir/tests/autoconf])])
+AS_IF([test -f $abs_top_builddir/tests/testsuite],
+ [AT_CHECK_SHELL_SYNTAX([$abs_top_builddir/tests/testsuite])])
# These are not built, they are in the src tree.
AT_CHECK_SHELL_SYNTAX([$abs_top_srcdir/config/install-sh])
@@ -336,7 +338,7 @@
#if VAL1
]])
-AT_CHECK([ifnames iftest1.c iftest2.c], 0,
+AT_CHECK([$ifnames iftest1.c iftest2.c], 0,
[DEF1 iftest1.c
DEF2 iftest1.c
DEF3 iftest1.c
@@ -558,17 +560,17 @@
AT_SETUP([autoupdating AC_PREREQ])
# Produce `AC_PREREQ(<AUTOUPDATE VERSION>)'.
-AT_CHECK([autoupdate --version | sed 's/.*) //;q'], 0, [stdout])
+AT_CHECK([$autoupdate --version | sed 's/.*) //;q'], 0, [stdout])
autoupdate_version=`cat stdout`
echo "AC_PREREQ($autoupdate_version)" >expout
-AT_CHECK([echo "AC_PREREQ(1.0)" | autoupdate -],
+AT_CHECK([echo "AC_PREREQ(1.0)" | $autoupdate -],
0, [expout], [])
-AT_CHECK([echo "AC_PREREQ($autoupdate_version)" | autoupdate -],
+AT_CHECK([echo "AC_PREREQ($autoupdate_version)" | $autoupdate -],
0, [expout], [])
-AT_CHECK([echo "AC_PREREQ(999.99)" | autoupdate -],
+AT_CHECK([echo "AC_PREREQ(999.99)" | $autoupdate -],
63, [], [ignore])
AT_CLEANUP
@@ -728,7 +730,7 @@
AT_SETUP([autoupdating with aclocal and m4@&address@hidden)
# We use aclocal.
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
+AT_CHECK([$aclocal --version || exit 77], [], [ignore], [ignore])
mkdir m4 aclocal
AT_DATA([configure.in],
@@ -743,7 +745,7 @@
AC_DEFUN([AC_UNCHANGED_MACRO], [echo one])
]])
cp m4/stuff.m4 aclocal/stuff.m4
-AT_CHECK([aclocal -I aclocal], [0], [ignore], [ignore])
+AT_CHECK_ACLOCAL([-I aclocal], [0], [ignore], [ignore])
# Checking `autoupdate'.
AT_CHECK_AUTOUPDATE
AT_CHECK([aclocal -I m4], [0], [ignore], [ignore])
Index: tests/torture.at
===================================================================
RCS file: /cvsroot/autoconf/autoconf/tests/torture.at,v
retrieving revision 1.60
diff -u -r1.60 torture.at
--- tests/torture.at 6 May 2006 05:53:50 -0000 1.60
+++ tests/torture.at 9 May 2006 16:31:40 -0000
@@ -769,7 +769,7 @@
AT_KEYWORDS(autoreconf)
# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [stdout], [ignore])
+AT_CHECK([$aclocal --version || exit 77], [], [stdout], [ignore])
# It should understand configure.ac.
AT_CHECK([[grep '1.[01234]' stdout && exit 77]], [1], [ignore])
@@ -814,7 +814,7 @@
# If there are improperly quoted AC_DEFUN installed in share/aclocal,
# they trigger warnings from aclocal 1.8, so ignore stderr.
-AT_CHECK([autoreconf], [], [], [ignore])
+AT_CHECK([$autoreconf], [], [], [ignore])
AT_CHECK([test -f inner/configure])
# Running the outer configure recursively should provide the innermost
@@ -865,7 +865,7 @@
AT_KEYWORDS(autoreconf)
# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
+AT_CHECK([$aclocal --version || exit 77], [], [ignore], [ignore])
# The contents of `.'
AT_DATA([install-sh], [])
@@ -894,7 +894,7 @@
AC_OUTPUT
]])
-AT_CHECK([autoreconf -Wall -v], [0], [ignore], [ignore])
+AT_CHECK([$autoreconf -Wall -v], [0], [ignore], [ignore])
AT_CHECK([test -f inner/configure])
AT_CHECK([test -f inner/innermost/configure])
AT_CHECK([test -f inner/innermost/config.hin])
@@ -929,7 +929,7 @@
AT_KEYWORDS([autoreconf])
# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
+AT_CHECK([$aclocal --version || exit 77], [], [ignore], [ignore])
AT_DATA([install-sh], [])
AT_DATA([configure.in],
@@ -950,7 +950,7 @@
]])
chmod +x inner/configure
-AT_CHECK([autoreconf -Wall -v], 0, [ignore], [ignore])
+AT_CHECK([$autoreconf -Wall -v], 0, [ignore], [ignore])
# Running the outer configure recursively should provide the innermost
# help strings.
@@ -972,12 +972,12 @@
AT_KEYWORDS([autoreconf])
# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
+AT_CHECK([$aclocal --version || exit 77], [], [ignore], [ignore])
# The test group directory is not necessarily _empty_, but it does not contain
# files meaningful to `autoreconf'.
-AT_CHECK([autoreconf -Wall -v], 1, [ignore], [ignore])
+AT_CHECK([$autoreconf -Wall -v], 1, [ignore], [ignore])
AT_CLEANUP
@@ -993,7 +993,7 @@
AT_KEYWORDS([autoreconf])
# We use aclocal and automake via autoreconf.
-AT_CHECK([automake --version || exit 77], [], [ignore], [ignore])
+AT_CHECK([$automake --version || exit 77], [], [ignore], [ignore])
AT_DATA([configure.in],
[[AC_INIT(GNU foo, 1.0)
@@ -1007,7 +1007,7 @@
AUTOMAKE_OPTIONS = foreign
]])
-AT_CHECK([autoreconf -Wall -v -i], 0, [ignore], [ignore])
+AT_CHECK([$autoreconf -Wall -v -i], 0, [ignore], [ignore])
AT_CHECK([test -f HeeHee.in])
AT_CLEANUP