bug#22533: Non-determinism in python-3 ".pyc" bytecode

[Ludovic Courtès]

Cyril Roelandt <address@hidden> skribis:

> Here is a version of the patch that works with the upstream Python, but
> that I cannot get to work with our Guix recipe.

At first sight the patch LGTM.  How does it not work for you? :-)

I applied this:

diff --git a/gnu/packages/patches/python-3-deterministic-build-info.patch 
b/gnu/packages/patches/python-3-deterministic-build-info.patch
index 22c372a..bdf9f20 100644
--- a/gnu/packages/patches/python-3-deterministic-build-info.patch
+++ b/gnu/packages/patches/python-3-deterministic-build-info.patch
@@ -15,3 +15,28 @@ We cannot pass it in CPPFLAGS due to whitespace in the DATE 
string.
  #ifndef DATE
  #ifdef __DATE__
  #define DATE __DATE__
+
+--- Lib/importlib/_bootstrap.py
++++ Lib/importlib/_bootstrap.py
+@@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics):
+         Implementing this method allows the loader to read bytecode files.
+         Raises IOError when the path cannot be handled.
+         """
+-        return {'mtime': self.path_mtime(path)}
++        return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH',
++                                               st.st_mtime))}
+ 
+     def _cache_bytecode(self, source_path, cache_path, data):
+         """Optional method which writes data (bytes) to a file path (a str).
+@@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader):
+     def path_stats(self, path):
+         """Return the metadata for the path."""
+         st = _path_stat(path)
+-        return {'mtime': st.st_mtime, 'size': st.st_size}
++        return {
++            'mtime':  float(_os.environ.get(b'SOURCE_DATE_EPOCH', 
st.st_mtime)),
++            'size': st.st_size
++        }
+ 
+     def _cache_bytecode(self, source_path, bytecode_path, data):
+         # Adapt between the two APIs

… and that leads to these test failures:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build address@hidden --rounds=2 -K

[...]

======================================================================
FAIL: test_bad_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 452, in test_bad_marshal
    self._test_bad_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 342, in _test_bad_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

======================================================================
FAIL: test_no_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 441, in test_no_marshal
    self._test_no_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 322, in _test_no_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

======================================================================
FAIL: test_non_code_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 445, in test_non_code_marshal
    self._test_non_code_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 331, in _test_non_code_marshal
    self.import_(file_path, '_temp')
AssertionError: ImportError not raised

======================================================================
FAIL: test_old_timestamp 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 471, in test_old_timestamp
    self.assertEqual(bytecode_file.read(4), source_timestamp)
AssertionError: b'\x01\x00\x00\x00' != b'\x7f\xc7\x04W'

======================================================================
FAIL: test_bad_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 452, in test_bad_marshal
    self._test_bad_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 342, in _test_bad_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

======================================================================
FAIL: test_no_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 441, in test_no_marshal
    self._test_no_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 322, in _test_no_marshal
    self.import_(file_path, '_temp')
AssertionError: EOFError not raised

======================================================================
FAIL: test_non_code_marshal 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 445, in test_non_code_marshal
    self._test_non_code_marshal()
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 331, in _test_non_code_marshal
    self.import_(file_path, '_temp')
AssertionError: ImportError not raised

======================================================================
FAIL: test_old_timestamp 
(test.test_importlib.source.test_file_loader.Source_SourceLoaderBadBytecodeTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/util.py",
 line 22, in wrapper
    to_return = fxn(*args, **kwargs)
  File 
"/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/test_importlib/source/test_file_loader.py",
 line 471, in test_old_timestamp
    self.assertEqual(bytecode_file.read(4), source_timestamp)
AssertionError: b'\x01\x00\x00\x00' != b'\x7f\xc7\x04W'

----------------------------------------------------------------------
Ran 951 tests in 1.102s

FAILED (failures=8, skipped=19, expected failures=1)
Makefile:958: recipe for target 'test' failed
--8<---------------cut here---------------end--------------->8---

‘test_old_timestamp’ clearly needs to be adjusted to account for the
change.  The others have to do with the bytecode loader, so it’s
probably a similar story.  Could you look into it?

Perhaps you tested with SOURCE_DATE_EPOCH unset?

Thanks for working on this, it’s an important bug to fix!

Ludo’.