Re: python debugging on trunk

[Zack Weinberg]

On Mon, Sep 26, 2022, at 12:23 PM, Karl Berry wrote:
> Is anyone up for debugging some Python-related test failures on
> RHEL-based systems?

I have access to a RHEL7 system, I know Python, and this sounds much
less unpleasant than everything else I'm supposed to be doing today.

> I have a suspicion the problem is that on RHEL systems, "python" invokes
> "python2" (because that's what the Python maintainers recommend, as I
> understand it), but the tests are assuming it invokes "python3".

Indeed:

$ lsb_release -d
Description: Red Hat Enterprise Linux Workstation release 7.9 (Maipo)
$ python --version
Python 2.7.5
$ python3 --version
Python 3.6.8

I have very strong opinions on this topic, having had to deal with a
lot of badly-written Python 2 code that, in the worst case, silently
corrupts data when run under Python 3.  IMNSHO, the bare command name
"python" and the absolute pathname "/usr/bin/python" MUST[rfc2119] be
*permanently reserved* for the Python 2 interpreter.  It's fine for
them not to exist at all, but if they exist and they invoke Python 3,
that is a *bug* in the distribution.

(Yes, there's a PEP that says something different -- IMO it is wrong,
its authors probably hadn't encountered the *really* troublesome
Python 2 code that I have.)

---
RHEL7, unsurprisingly, ships with Autoconf 2.69.  Before attempting to
build Automake from Git, I built Autoconf 2.71 from the tarball
release, installed it in a subdirectory of my homedir, and made that
directory come first on PATH.  There were no unexpected test failures
in Autoconf's testsuite. Other relevant tool versions include:

$ perl --version
This is perl 5, version 16, subversion 3 (v5.16.3) built for 
x86_64-linux-thread-multi
(with 44 registered patches, see perl -V for more detail)

$ m4 --version
m4 (GNU M4) 1.4.16

$ /bin/sh --version
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

$ make --version
GNU Make 3.82

$ sort --version
sort (GNU coreutils) 8.22

$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)

$ libtool --version
libtool (GNU libtool) 2.4.2

I see these Automake testsuite failures:

Testsuite summary for GNU Automake 1.16i
========================================
# TOTAL: 2988
# PASS:  2877
# SKIP:  54
# XFAIL: 39
# FAIL:  18
# XPASS: 0
# ERROR: 0

FAIL: t/canon7
FAIL: t/fort5
FAIL: t/instmany-python
FAIL: t/ltinit
FAIL: t/nobase-python
FAIL: t/pr401b
FAIL: t/py-compile-basedir
FAIL: t/py-compile-basic
FAIL: t/py-compile-destdir
FAIL: t/py-compile-option-terminate
FAIL: t/python-pr10995
FAIL: t/python-prefix
FAIL: t/python-vars
FAIL: t/python10
FAIL: t/python12
FAIL: t/python3
FAIL: t/subobj9
FAIL: t/suffix5

That's _most_ of the failures you got, plus some more:

+FAIL: t/canon7
+FAIL: t/fort5
+FAIL: t/ltinit
+FAIL: t/pr401b
+FAIL: t/python10
+FAIL: t/subobj9
+FAIL: t/suffix5
-FAIL: t/subobj-pr13928-more-langs

There are only two failure modes:

canon7, fort5, ltinit, pr401b, subobj9, suffix5

ERROR: files left in build directory after distclean:
./.__afsxxxx

I did the build in an AFS volume; these are the AFS equivalent of
the NFS "silly rename" hack (apparently it can happen with SMB as well).
"make distcleancheck" should ignore these.

instmany-python, nobase-python, py-compile-basic, py-compile-destdir,
py-compile-option-terminate, pyghon-pr10995, python-prefix,
python-vars, python10, python12, python3

AttributeError: 'module' object has no attribute 'implementation'

This is coming from this code in lib/py-compile (there are three
copies of it):

    if hasattr(sys.implementation, 'cache_tag'):
        py_compile.compile(filepath, 
importlib.util.cache_from_source(filepath), path)
    else:
        py_compile.compile(filepath, filepath + 'c', path)

sys.implementation was added in Python 3.3 and
importlib.util.cache_from_source was added in 3.4.
Python 2.7's importlib _only_ has 'import_module'.

The appended patch should address both issues.  Note that I have only
tested it with Python 2.7 and 3.6.  It _may_ not be correct for Python
in the 3.0--3.3 (inclusive) range; I cannot conveniently test that.

zw

  * lib/am/distdir.am (distcleancheck_listfiles): Filter out "silly rename"
  files, unavoidably created by deleting files that are still open in some
  process on various network file systems.

  * lib/py-compile: Test directly for availability of
  importlib.util.cache_from_source.  Untangle logic for when to generate
  -O and -OO bytecode.  Reformat embedded Python fragments.

diff --git a/lib/am/distdir.am b/lib/am/distdir.am
index 0f591266d..b8d90e572 100644
--- a/lib/am/distdir.am
+++ b/lib/am/distdir.am
@@ -568,7 +568,9 @@ distuninstallcheck:
 ## Define distcleancheck_listfiles and distcleancheck separately
 ## from distcheck, so that they can be overridden by the user.
 .PHONY: distcleancheck
-distcleancheck_listfiles = find . -type f -print
+distcleancheck_listfiles = \
+  find . \( -type f -a \! \
+  \( -name .nfs* -o -name .smb* -o -name .__afs* \) \) -print
 distcleancheck: distclean
        @if test '$(srcdir)' = . ; then \
          echo "ERROR: distcleancheck can only run from a VPATH build" ; \
diff --git a/lib/py-compile b/lib/py-compile
index 24a8a1a54..de0f02e32 100755
--- a/lib/py-compile
+++ b/lib/py-compile
@@ -1,7 +1,7 @@
 #!/bin/sh
 # py-compile - Compile a Python program
 
-scriptversion=2022-02-06.07; # UTC
+scriptversion=2022-09-26.22; # UTC
 
 # Copyright (C) 2000-2022 Free Software Foundation, Inc.
 
@@ -141,88 +141,83 @@ python_minor=`$PYTHON -c 'import sys; 
print(sys.version_info[1])'`
 $PYTHON -c "
 import sys, os, py_compile, importlib
 
+# importlib.util.cache_from_source was added in 3.4
+if (
+        hasattr(importlib, 'util')
+        and hasattr(importlib.util, 'cache_from_source')
+):
+    destpath = importlib.util.cache_from_source
+else:
+    destpath = lambda filepath: filepath + 'c'
+
 sys.stdout.write('Byte-compiling python modules...\n')
 for file in sys.argv[1:]:
     $pathtrans
     $filetrans
-    if not os.path.exists(filepath) or not (len(filepath) >= 3
-                                            and filepath[-3:] == '.py'):
-           continue
+    if (
+            not os.path.exists(filepath)
+            or not (len(filepath) >= 3 and filepath[-3:] == '.py')
+     ):
+        continue
     sys.stdout.write(file + ' ')
     sys.stdout.flush()
-    if hasattr(sys.implementation, 'cache_tag'):
-        py_compile.compile(filepath, 
importlib.util.cache_from_source(filepath), path)
-    else:
-        py_compile.compile(filepath, filepath + 'c', path)
+    py_compile.compile(filepath, destpath(filepath), path)
 sys.stdout.write('\n')" "$@" || exit $?
 
 # Then byte compile w/optimization all the modules.
-case $python_major.$python_minor in
-2.*)
-  $PYTHON -O -c "
+$PYTHON -O -c "
 import sys, os, py_compile, importlib
 
-# pypy does not use .pyo optimization
-if hasattr(sys, 'pypy_translation_info'):
+# importlib.util.cache_from_source was added in 3.4
+if (
+        hasattr(importlib, 'util')
+        and hasattr(importlib.util, 'cache_from_source')
+):
+    destpath = importlib.util.cache_from_source
+else:
+    destpath = lambda filepath: filepath + 'o'
+
+# pypy2 does not use .pyo optimization
+if sys.version_info.major <= 2 and hasattr(sys, 'pypy_translation_info'):
     sys.exit(0)
 
 sys.stdout.write('Byte-compiling python modules (optimized versions) ...\n')
 for file in sys.argv[1:]:
     $pathtrans
     $filetrans
-    if not os.path.exists(filepath) or not (len(filepath) >= 3
-                                            and filepath[-3:] == '.py'):
-           continue
+    if (
+            not os.path.exists(filepath)
+            or not (len(filepath) >= 3 and filepath[-3:] == '.py')
+    ):
+        continue
     sys.stdout.write(file + ' ')
     sys.stdout.flush()
-    if hasattr(sys.implementation, 'cache_tag'):
-        py_compile.compile(filepath, 
importlib.util.cache_from_source(filepath), path)
-    else:
-        py_compile.compile(filepath, filepath + 'o', path)
+    py_compile.compile(filepath, destpath(filepath), path)
 sys.stdout.write('\n')" "$@" 2>/dev/null || exit $?
-  ;;
-*)  # Python 3+
-  $PYTHON -O -c "
-import sys, os, py_compile, importlib
-
-print('Byte-compiling python modules (optimized versions) ...')
-for file in sys.argv[1:]:
-    $pathtrans
-    $filetrans
-    if not os.path.exists(filepath) or not (len(filepath) >= 3
-                                            and filepath[-3:] == '.py'):
-           continue
-    print(file, end=' ', flush=True)
-    if hasattr(sys.implementation, 'cache_tag'):
-        py_compile.compile(filepath, 
importlib.util.cache_from_source(filepath), path)
-    else:
-        py_compile.compile(filepath, filepath + 'o', path)
-print()" "$@" 2>/dev/null || exit $?
-  ;;
-esac
 
 # Then byte compile w/more optimization.
+# Only do this for Python 3.5+, see https://bugs.gnu.org/38043 for background.
 case $python_major.$python_minor in
 2.*|3.[0-4])
   ;;
-*)  # Python 3.5+
-  # See https://bugs.gnu.org/38043 for background.
+*)
   $PYTHON -OO -c "
-import sys, os, py_compile, imp
+import sys, os, py_compile, importlib
 
-print('Byte-compiling python modules (-OO version) ...')
+sys.stdout.write('Byte-compiling python modules (more optimized versions)'
+                 ' ...\n')
 for file in sys.argv[1:]:
     $pathtrans
     $filetrans
-    if not os.path.exists(filepath) or not (len(filepath) >= 3
-                                            and filepath[-3:] == '.py'):
+    if (
+            not os.path.exists(filepath)
+            or not (len(filepath) >= 3 and filepath[-3:] == '.py')
+    ):
         continue
-    print(file, end=' ', flush=True)
-    if hasattr(imp, 'get_tag'):
-        py_compile.compile(filepath, imp.cache_from_source(filepath), path)
-    else:
-        py_compile.compile(filepath, filepath + 'o', path)
-print()" "$@" 2>/dev/null || exit $?
+    sys.stdout.write(file + ' ')
+    sys.stdout.flush()
+    py_compile.compile(filepath, importlib.util.cache_from_source(filepath), 
path)
+sys.stdout.write('\n')" "$@" 2>/dev/null || exit $?
   ;;
 esac
 
--end--