branch master updated: website: Add draft post about virtual build machines.

[Ludovic Courtès]

This is an automated email from the git hooks/post-receive script.

civodul pushed a commit to branch master
in repository guix-artwork.

The following commit(s) were added to refs/heads/master by this push:
     new 819aee1  website: Add draft post about virtual build machines.
819aee1 is described below

commit 819aee16726e7adcc100b49ed2102a741c79320a
Author: Ludovic Courtès <ludo@gnu.org>
AuthorDate: Sun Mar 10 22:45:24 2024 +0100

    website: Add draft post about virtual build machines.
    
    * website/drafts/build-vm.md, website/static/blog/img/safety-last.jpg:
    New files.
---
 website/drafts/build-vm.md              | 386 ++++++++++++++++++++++++++++++++
 website/static/blog/img/safety-last.jpg | Bin 0 -> 144385 bytes
 2 files changed, 386 insertions(+)

diff --git a/website/drafts/build-vm.md b/website/drafts/build-vm.md
new file mode 100644
index 0000000..fd6aadc
--- /dev/null
+++ b/website/drafts/build-vm.md
@@ -0,0 +1,386 @@
+title: Adventures on the quest for long-term reproducible deployment
+author: Ludovic Courtès
+tags: Reproducibility, Research
+date: 2024-03-13 14:00
+---
+
+Rebuilding software five years later, how hard can it be?  It can’t be
+*that* hard, especially when you pride yourself on having a tool that
+can [travel in
+time](https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-time_002dmachine.html)
+and that does a good job at ensuring [reproducible
+builds](https://reproducible-builds.org/docs/definition/), right?
+
+Well, in hindsight, we can tell you: it’s more challenging than it
+seems.  Users attempting traveling 5 years back with `guix time-machine`
+are (or *were*) unavoidably going to hit bumps on the road—a real
+problem because that’s one of the use cases Guix aims to support well,
+in particular in a [reproducible
+research](https://hpc.guix.info/blog/tag/reproducibility/) context.
+
+In this post, we look at some of the challenges we face while traveling
+back, how we are overcoming them, and open issues.
+
+# The vision
+
+First of all, one clarification: Guix aims to support time travel, but
+we’re talking of a time scale measured in years, not in decades.  We
+know all too well that this is already very ambitious—it’s something
+that probably nobody except [Nix](https://nixos.org) and Guix are even
+trying.  More importantly, software deployment at the scale of decades
+calls for very different, more radical techniques; it’s the work of
+archivists.
+
+Concretely, Guix 1.0.0 was [released in
+2019](https://guix.gnu.org/en/blog/2019/gnu-guix-1.0.0-released/) and
+our goal is to allow users to travel as far back as 1.0.0 and redeploy
+software from there, as in this example:
+
+```
+guix time-machine -q --commit=v1.0.0 -- install python2
+```
+
+It’s only 5 years ago but it’s pretty much remote history on the scale
+of software evolution.  How well does such a command work?  Well, it
+depends.
+
+The project has two build farms; `bordeaux.guix.gnu.org` has been
+keeping substitutes (pre-built binaries) of everything it built since
+roughly 2021, while `ci.guix.gnu.org` keeps substitutes for roughly two
+years.  Time traveling to a period where substitutes are available is
+fine: you end up downloading lots of binaries, but that’s OK, you rather
+quickly have your software environment at hand.
+
+# Bumps on the build road
+
+Things get more complicated when targeting a period in time for which
+substitutes are no longer available, such as `v1.0.0` above.  (And
+really, we should assume that substitutes won’t remain available
+forever: fellow NixOS hackers recently had to seriously consider
+[trimming their 20-year-long history of
+substitutes](https://discourse.nixos.org/t/nixos-s3-long-term-resolution-phase-1/36493)
+because the costs are not sustainable.)
+
+The obvious first problem that arises in the absence of substitutes is
+source code unavailability.  I’ll spare you the details for this
+post—that problem alone would deserve a book.  Suffice to say that we’re
+lucky that we started working on [integrating Guix with Software
+Heritage](https://guix.gnu.org/en/blog/2019/connecting-reproducible-deployment-to-a-long-term-source-code-archive/)
+years ago, and that there has been great progress over the last couple
+of years to get closer to [full package source code
+archival](https://ngyro.com/pog-reports/latest/) (more precisely: 94% of
+the source code of packages available in Guix in January 2024 is
+archived, versus 72% of the packages available in May 2019).
+
+So what happens when you run the `time-machine` command above?  It
+brings you to May 2019, a time for which none of the official build
+farms had substitutes until a few days ago.  Ideally, thanks to
+[isolated build
+environments](https://guix.gnu.org/manual/devel/en/html_node/Build-Environment-Setup.html),
+you’d build things for hours or days, and in the end all those binaries
+will be here just as they were 5 years ago.  In practice though, there
+are several problems that isolation as currently implemented does *not*
+address.
+
+![Screenshot of movie “Safety Last!” with Harold Lloyd hanging from a clock on 
a building’s façade.](/static/blog/img/safety-last.jpg)
+
+Among those, the most frequent problem is *time traps*: software build
+processes that fail after a certain date (these are also referred to as
+“time bombs” but we’ve had enough of these and would rather call for a
+ceasefire).  This plagues less than 1% of the package collection, but
+unfortunately we’re talking about packages deep in the dependency graph.
+Here are some examples:
+
+  - [OpenSSL](https://issues.guix.gnu.org/56137) test suite failures
+       after a certain date because some of the X.509 certificates used in
+       its tests have expired.
+  - Similar issue with [GnuTLS](https://issues.guix.gnu.org/44559).
+       Newer upstream versions rely on
+       [datefudge](https://packages.guix.gnu.org/packages/datefudge/) to
+       fake the date while running the tests and thus avoid that problem
+       altogether.
+  - Python 2.7, found in Guix 1.0.0, had a [similar
+       issue](https://issues.guix.gnu.org/65378) with its TLS-related
+       tests.
+  - OpenJDK [would fail to build at some
+       point](https://issues.guix.gnu.org/68333) with this interesting
+       message: `Error: time is more than 10 years from present:
+       1388527200000` (the build system would consider that its data about
+       currencies is likely outdated after 10 years).
+  - Libgit2, a dependency of Guix, had (has?) a [time-dependent
+       tests](https://issues.guix.gnu.org/55326).
+  - MariaDB tests [started failing after January 1st,
+       2019](https://issues.guix.gnu.org/34351).
+
+Someone traveling to `v1.0.0` will hit several of these, preventing
+`guix time-machine` from completing.  A serious bummer, especially to
+those who’ve come to Guix from the perspective of making their [research
+workflow
+reproducible](https://hpc.guix.info/blog/2023/06/a-guide-to-reproducible-research-papers/).
+
+Time traps are the main road block, but there’s more!  Occasionally,
+there’s software influenced by kernel details not controlled by the
+build daemon:
+
+  - Tests of the hwloc hardware locality library [would fail when
+       running on a Btrfs file system](https://issues.guix.gnu.org/54767).
+
+In a handful of cases, but important ones, builds might fail when
+performed on certain CPUs.  We’re aware of at least two cases:
+
+  - Python 3.9 to 3.11 would set a signal handler stack [too small for
+       use on Intel Sapphire Rapids Xeon
+       CPUs](https://github.com/python/cpython/issues/91124) (it’s more
+       complicated than this but the end result is: it will no longer build
+       on modern hardware).
+  - Firefox would reportedly [crash on Raptor Lake CPUs running an buggy
+       version of their
+       firmware](https://bugzilla.mozilla.org/show_bug.cgi?id=1882015).
+
+Neither time traps nor those obscure hardware-related issues can be
+avoided with the isolation mechanism currently used by the build daemon.
+This harms time traveling when substitutes are unavailable.  Giving up
+is not in the ethos of this project though.
+
+# Where to go from here?
+
+There are really two open questions here:
+
+  1. How can we tell which packages needs to be “fixed”, and how:
+        building at a specific date, on a specific CPU?
+  2. How can keep those aspects of the build environment (time, CPU
+        variant) under control?
+
+Let’s start with #2.  Before looking for a solution, it’s worth
+remembering where we come from.  The build daemon runs build processes
+with a [separate root file
+system](https://www.man7.org/linux/man-pages/man2/chroot.2.html), under
+dedicated user IDs, and in separate [Linux
+namespaces](https://www.man7.org/linux/man-pages/man7/namespaces.7.html),
+thereby minimizing interference with the rest of the system and ensuring
+a [well-defined build
+environment](https://guix.gnu.org/manual/devel/en/html_node/Build-Environment-Setup.html).
+This technique was
+[implemented](https://archive.softwareheritage.org/browse/revision/9397cd30c8a6ffd65fc3b85985ea59ecfb72672b/)
+by Eelco Dolstra *et al.* for Nix in 2007 (with namespace support [added
+in
+2012](https://archive.softwareheritage.org/browse/revision/df716c98d203ab64cdf05f9c17fdae565b7daa1c/)),
+at a time where the word *container* had to do with boats and before
+“Docker” became the name of a software tool.  In short, the approach
+consists in *controlling the build environment* in every detail (it’s at
+odds with the strategy that consists in achieving reproducible builds
+[*in spite* of high build environment
+variability](https://tests.reproducible-builds.org/debian/index_variations.html)).
+That these are mere processes with a bunch of bind mounts makes build
+processes rather inexpensive.
+
+Thus, naturally, we’d want to control the build environment’s date, and
+naturally, we turn to Linux namespaces to address that—Dolstra, Löh, and
+Pierron already suggested something along these lines in the conclusion
+of their [2010 *Journal of Functional Programming*
+paper](https://edolstra.github.io/pubs/nixos-jfp-final.pdf).  Turns out
+there *is* now a [time
+namespace](https://www.man7.org/linux/man-pages/man7/time_namespaces.7.html).
+Unfortunately it’s limited to `CLOCK_MONOTONIC` and `CLOCK_BOOTTIME`
+clocks; the manual page states:
+
+> Note that time namespaces do not virtualize the `CLOCK_REALTIME`
+> clock.  Virtualization of this clock was avoided for reasons of
+> complexity and overhead within the kernel.
+
+I hear you say: *What about
+[datefudge](https://packages.guix.gnu.org/packages/datefudge/) and
+[libfaketime](https://packages.guix.gnu.org/packages/libfaketime/)?*
+These rely on the `LD_PRELOAD` environment variable to trick the dynamic
+linker into pre-loading a library that provides symbols such as
+`gettimeofday` and `clock_gettime`.  This is a fine approach in some
+cases, but it’s too fragile and too intrusive when targeting arbitrary
+build processes.
+
+That leaves us with essentially one viable option: virtual machines
+(VMs).  The full-system QEMU lets you specify the initial real-time
+clock of the VM with the `-rtc` flag, which is exactly what we need
+(“user-land” QEMU such as `qemu-x86_64` does not support it).  And of
+course, it lets you specify the CPU model to emulate.
+
+# News from the past
+
+Now, the question is: where does the VM fit?  The author considered
+writing a [package
+transformation](https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html)
+that would change a package such that it’s built in a well-defined VM.
+However, that wouldn’t really help: this option didn’t exist in past
+revisions, and it would lead to a different build anyway from the
+perspective of the daemon—a different
+[*derivation*](https://guix.gnu.org/manual/devel/en/html_node/Derivations.html).
+
+The best strategy appeared to be
+[*offloading*](https://guix.gnu.org/manual/devel/en/html_node/Daemon-Offload-Setup.html):
+the build daemon can offload builds to different machines over SSH, we
+just need to let it send builds to a suitably-configured VM.  To do
+that, we can reuse some of the machinery initially developed for
+[*childhurds*](https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-childhurd_002c-offloading)
+that takes care of setting up offloading to the VM: creating substitute
+signing keys and SSH keys, exchanging secret key material between the
+host and the guest, and so on.
+
+The end result is a [Guix System
+service](https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#Virtual-Build-Machines)
+that can be configured in a few lines:
+
+```scheme
+(use-modules (gnu services virtualization))
+
+(operating-system
+  ;; …
+  (services (append (list (service virtual-build-machine-service-type))
+                                       %base-services)))
+```
+
+The default setting above provides a 4-core VM whose initial date is
+January 2020, emulating a Skylake CPU from that time—the right setup for
+someone willing to reproduce old binaries.  You can check the
+configuration like this:
+
+```
+$ sudo herd configuration build-vm
+CPU: Skylake-Client
+number of CPU cores: 4
+memory size: 2048 MiB
+initial date: Wed Jan 01 00:00:00Z 2020
+```
+
+To enable offloading to that VM, one has to explicitly start it, like
+so:
+
+```
+$ sudo herd start build-vm
+```
+
+From there on, every native build is offloaded to the VM.  The key part
+is that with almost no configuration, you get everything set up to build
+packages “in the past”.  It’s a Guix System only solution; if you run
+Guix on another distro, you can set up a similar build VM but you’ll
+have to go through the cumbersome process that is all taken care of
+automatically here.
+
+Of course it’s possible to choose different configuration parameters:
+
+```scheme
+(service virtual-build-machine-service-type
+                (virtual-build-machine
+                 (date (make-date 0 0 00 00 01 10 2017 0)) ;further back in 
time
+                 (cpu "Westmere")
+                 (cpu-count 16)
+                 (memory-size (* 8 1024))
+                 (auto-start? #t)))
+```
+
+With a build VM with its date set to January 2020, we have been able to
+rebuild Guix and its dependencies along with a bunch of packages such as
+`emacs-minimal` from `v1.0.0`, overcoming all the time traps and other
+pleasant challenges described earlier.  As a side effect, substitutes
+are now available from `ci.guix.gnu.org` so you can even try this at
+home without having to rebuild the world:
+
+```
+$ guix time-machine -q --commit=v1.0.0 -- build emacs-minimal --dry-run
+guile: warning: failed to install locale
+substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
+38.5 MB would be downloaded:
+   /gnu/store/53dnj0gmy5qxa4cbqpzq0fl2gcg55jpk-emacs-minimal-26.2
+```
+
+For the fun of it, we went as far as
+[`v0.16.0`](https://guix.gnu.org/blog/2018/gnu-guix-and-guixsd-0.16.0-released/),
+released in December 2018:
+
+```
+guix time-machine -q --commit=v0.16.0 -- \
+  environment --ad-hoc vim -- vim --version
+```
+
+This is the furthest we can go since
+[channels](https://guix.gnu.org/manual/devel/en/html_node/Channels.html)
+and the underlying mechanisms that make time travel possible did not
+exist before that date.
+
+There’s at least one case where things got more complicated as we tried
+to build packages from these revisions: in OpenSSL 1.1.1g (released
+April 2020 and packaged [in December
+2020](https://archive.softwareheritage.org/browse/revision/c4868e38289baf3a9a74bdf32166d321f7365725/)),
+some of the test certificates are not valid _before_ April 2020, so the
+build VM needs to have its clock set to May 2020 or thereabouts.
+Booting the build VM with a different date can be done without
+reconfiguring the system:
+
+```
+$ sudo herd stop build-vm
+$ sudo herd start build-vm -- -rtc base=2020-05-01T00:00:00
+```
+
+The `-rtc …` flags are passed straight to QEMU, which is handy when
+exploring workarounds…
+
+The [`time-travel` continuous integration
+jobset](https://ci.guix.gnu.org/jobset/time-travel) has been set up to
+check that we can, at any time, travel back to one of the past releases.
+This at least ensures that Guix itself and its dependencies have
+substitutes available at `ci.guix.gnu.org`.
+
+# Reproducible research workflows reproduced
+
+Incidentally, this effort rebuilding 5-year-old packages has allowed us
+to fix embarrassing problems.  Software that accompanies “reproducible”
+research papers could no longer be deployed, at least not without this
+clock twiddling effort:
+
+  - 
[code](https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://gitlab.inria.fr/lcourtes-phd/edcc-2006-redone)
+       of [_[Re] Storage Tradeoffs in a Collaborative Backup Service for
+       Mobile Devices_](https://doi.org/10.5281/zenodo.3886739), submitted
+       as part of the ReScience [_Ten Years Reproducibility
+       Challenge_](https://rescience.github.io/ten-years/) in June 2020,
+       and which is precisely about showcasing reproducible deployment with
+       Guix;
+  - 
[code](https://archive.softwareheritage.org/browse/revision/707f00afef8f6ef1f29a7a4c961dd714f82833f5/)
+       of the 2022 Nature Scientific Data article entitled [_Toward
+       practical transparent verifiable and long-term reproducible research
+       using Guix_](https://doi.org/10.1038/s41597-022-01720-9), which
+       relied on an April 2020 revision of Guix to deploy (Simon Tournier
+       who co-authored the paper [reported
+       earlier](https://simon.tournier.info/posts/2023-12-21-repro-paper.html)
+       on a failed attempt showing just how challenging it was).
+
+It’s good news that we can now re-deploy these 5-year-old software
+environments with minimum hassle; it’s bad news that holding this
+promise took extra effort.
+
+# The future
+
+The astute reader surely noticed that we didn’t answer question #1
+above:
+
+> How can we tell which packages needs to be “fixed”, and how: building
+> at a specific date, on a specific CPU?
+
+It’s a fact that Guix so far lacks information about the date, kernel,
+or CPU model that should be used to build a given package.
+[Derivations](https://guix.gnu.org/manual/devel/en/html_node/Derivations.html)
+purposefully lack that information on the grounds that it’s *rarely*
+necessary—which is true, but “rarely” is not the same as “never”, as we
+saw.  Should we start adding such annotations to packages?
+
+Here’s another option: build packages in VMs running in the year 2100,
+say, and on a baseline CPU.  We don’t necessarily need to require all
+users to set up a virtual build machine, it may be enough to set up the
+project build farms so they build everything that way.  This would allow
+us to catch time traps and Y2038 bugs before they bite.
+
+Before we can get there the `virtual-build-machine` service needs to be
+optimized.  Right now, offloading to build VMs is as heavyweight as
+offloading to a separate physical build machine: data is transferred
+back and forth over SSH over TCP/IP.  The first step will be to run SSH
+over a paravirtualized transport instead such as [`AF_VSOCK`
+sockets](https://www.man7.org/linux/man-pages/man7/vsock.7.html).
+Another option would be to make the guest VM store an overlay over the
+host VM store such that inputs do not need to be transferred and copied.
diff --git a/website/static/blog/img/safety-last.jpg 
b/website/static/blog/img/safety-last.jpg
new file mode 100644
index 0000000..9ccbd4b
Binary files /dev/null and b/website/static/blog/img/safety-last.jpg differ