--- Begin Message ---
|
Subject: |
Non-determinism in python-3 ".pyc" bytecode |
|
Date: |
Tue, 2 Feb 2016 00:15:44 -0500 |
|
User-agent: |
Mutt/1.5.24 (2015-08-30) |
While preparing a package for borg [0], I found that the built output
was not reproducible. The problem is that the bytecode compiler [1] for
Python 3.4.3 (our current version) encodes the mtime of the
corresponding Python source file in the output. This is described in
PEP-3147 [2], and the responsible Python code is referenced below [3].
I tested a few of our existing python-3 packages: python-ccm,
python-pysam, and python-scripttest all exhibit the same problem.
We fixed this in python-2 with the patch
python-2.7-source-date-epoch.patch, but I don't know how to write this
patch for python-3.
Can somebody write this patch?
I asked about this on #debian-reproducible and they said that it wasn't
an issue for Debian since they don't ship bytecode, but instead generate
it at install time. Of course, that doesn't really apply to Guix.
I used diffoscope-34 to inspect the build outputs to find this, and you
can see the report here:
https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html
It's first demonstrated in the file
...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.
The first 2 bytes are the "magic numbers" described in PEP-3147, which
specify the version of the bytecode format. The next 2 bytes are the
problematic timestamp, as described in the PEP-3147.
[0]
http://borgbackup.github.io/
[1]
https://docs.python.org/3/library/py_compile.html
[2]
https://www.python.org/dev/peps/pep-3147/
[3] Check out the Guix git commit 4efc8eb27502c, and from there:
$ tar xf $(./pre-inst-env guix build --source python-3)
$ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
bytecode = importlib._bootstrap._code_to_bytecode(
code, source_stats['mtime'], source_stats['size'])
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#22533: Python bytecode reproducibility |
|
Date: |
Sun, 03 Feb 2019 22:22:23 +0100 |
|
User-agent: |
mu4e 1.0; emacs 26.1 |
Ricardo Wurmus <address@hidden> writes:
> Now that we’re using Python 3.7 and this version supports hash-based pyc
> files, is this still an issue? Do we need to do anything to enable
> hash-based pyc compilation?
>
> See:
> https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files
> https://www.python.org/dev/peps/pep-0552/
It looks like this is no longer a problem. I built borg just now and
the pyc files are reproducible.
(The man pages include a date stamp, though, which I’m trying to patch
now.)
--
Ricardo
--- End Message ---