[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
branch master updated: website: grafts-continued: Improve.
|
From: |
Ludovic Courtčs |
|
Subject: |
branch master updated: website: grafts-continued: Improve. |
|
Date: |
Wed, 06 May 2020 07:03:24 -0400 |
This is an automated email from the git hooks/post-receive script.
civodul pushed a commit to branch master
in repository guix-artwork.
The following commit(s) were added to refs/heads/master by this push:
new 9a4b927 website: grafts-continued: Improve.
9a4b927 is described below
commit 9a4b927ce2aa3844ea3b04fdc1c615d4a4e1666e
Author: Ludovic Courtès <address@hidden>
AuthorDate: Wed May 6 12:52:08 2020 +0200
website: grafts-continued: Improve.
* website/drafts/grafts-continued.md: Fix typo, improve wording. Add
derivation graph. Add sample session with 'package-derivation'.
Augment conclusion.
* website/static/blog/img/inkscape-graft.dot,
website/static/blog/img/inkscape-graft.svg: New files.
---
website/drafts/grafts-continued.md | 80 ++++++++++++++++++++++--------
website/static/blog/img/inkscape-graft.dot | 17 +++++++
website/static/blog/img/inkscape-graft.svg | 79 +++++++++++++++++++++++++++++
3 files changed, 155 insertions(+), 21 deletions(-)
diff --git a/website/drafts/grafts-continued.md
b/website/drafts/grafts-continued.md
index b45ec11..db58d4f 100644
--- a/website/drafts/grafts-continued.md
+++ b/website/drafts/grafts-continued.md
@@ -1,4 +1,4 @@
-title: Grafts, the continuation
+title: Grafts, continued
author: Ludovic Courtès
tags: Functional programming, Scheme API
--
@@ -8,7 +8,7 @@ users with [security
updates](https://guix.gnu.org/manual/en/html_node/Security-Updates.html)
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
-unavoidably stroke by what turned out to be the undesirable side effect
+unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but [1.1.0 finally addressed these
issues](https://guix.gnu.org/blog/2020/gnu-guix-1.1.0-released/).
@@ -52,7 +52,7 @@ potentially applicable grafts. Why “potentially applicable”?
Consider
this scenario: assume `perl` has a `replacement`; `coreutils` has a
dependency on `perl`, but it’s a build-time dependency: `coreutils` does
not depend on `perl` at run time. Thus, `coreutils` can be used as is,
-there is no need to try to graft it.
+there is no need to graft it.
But how do we know whether a dependency is a built-time-only dependency?
The [`native-inputs`
@@ -61,10 +61,11 @@ of a package usually lists build-time dependencies, but
it’s more of
hint. Ultimately, the set of run-time dependencies, which we call the
_references_, is the subset of the build-time dependencies that the
garbage collector (GC) in the build daemon finds _in the build
-result_—Section 5.5.1 of [Eelco Dolstra’s PhD thesis on
-Nix](http://nixos.org/~eelco/pubs/phd-thesis.pdf) describes how the GC
+result_—Section 5.5.1 of [Eelco Dolstra’s PhD
+thesis](http://nixos.org/~eelco/pubs/phd-thesis.pdf) describes how the
+GC
scans for references. In our example, we first have to actually build
-`coreutils` before we can tell whether or not it depends on `perl` at
+`coreutils` before we can tell whether it depends on `perl` at
run time.
Guix arranges to graft only when necessary. In this example, `guix
@@ -72,7 +73,11 @@ build coreutils` would return the same as `guix build
coreutils
--no-grafts`. Conversely, since `inkscape` has a run-time dependency on
`perl`, `guix build inkscape` returns a derivation that grafts the
`perl` replacement onto the original `inkscape` build result, the one
-returned by `guix build inkscape --no-grafts`.
+returned by `guix build inkscape --no-grafts`. The (simplified)
+dependency graph of the derivation for the grafted `inkscape` looks like
+this:
+
+
Grafts are a form of what [_Build Systems Ă la
Carte_](https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf)
@@ -146,7 +151,7 @@ as specified by
# Gathering dynamic dependencies
To address this, all these individual dynamic dependencies need to be
-gathered somehow instead of being treated one-by-one. Conceptually, we
+gathered somehow instead of being treated one by one. Conceptually, we
would like to, roughly, do a first pass lowering packages to derivations
as if grafting was disabled, build all these derivations, and then do a
second pass determine which packages in the graph need to be grafted and
@@ -159,7 +164,27 @@ the “ungrafted” one followed by the grafted one.
The problem is that our API is inherently serial: the
`package-derivation` function takes _one_ package, lowers it, and
-returns its derivation. Lowering includes dealing with grafts, and
+returns its derivation:
+
+```scheme
+(use-modules (guix)
+ (gnu packages base)
+ (gnu packages inkscape))
+
+(define s (open-connection))
+
+(package-derivation s coreutils)
+⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv
=> /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug
/gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
+(package-derivation s coreutils #:graft? #f)
+⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv
=> /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug
/gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
+
+(package-derivation s inkscape)
+⇒ #<derivation /gnu/store/jzm2zsq8m0rj8wdsmikc0p2ik0cprrcf-inkscape-0.92.4.drv
=> /gnu/store/clj8rjnsip8a35hyd9nf4l65w7ahn0gs-inkscape-0.92.4 7f6c9c15b730>
+(package-derivation s inkscape #:graft? #f)
+⇒ #<derivation /gnu/store/psd31x1fq0v2g594z217mh56xzg21dym-inkscape-0.92.4.drv
=> /gnu/store/zz28ckjwfxwkx3gsm8sc452kmvfiky6y-inkscape-0.92.4 7f6c90ad4f50>
+```
+
+Lowering includes dealing with grafts, and
that’s why we ended up with one-by-one inefficiencies. An option would
be to make all the API “plural”: have `package-derivation` and its
friends accept a _list_ of packages instead of a single one. That would
@@ -246,8 +271,9 @@ What we see above is first a build plan that downloads
binaries for the
two ungrafted packages, followed by a build plan for one grafting
derivations: we have successfully preserved parallelism.
-The solution is not as principled as the excellent decomposition the _Ă
-la Carte_ paper proposes. It remains an approximation and not the
+The solution resembles the `suspending` scheduler discussed in the _Ă
+la Carte_ paper, though decomposition is not as principled as what the
+paper describes. It remains an approximation and not the
optimal way to deal with dynamic dependencies. There are still
situations [where that shows](https://issues.guix.gnu.org/issue/40612),
but overall, it’s a significant improvement. Unlike [other solutions
@@ -286,8 +312,8 @@ welcome ideas!
# Dynamic dependencies of all shapes
We have seen how Guix deals with dynamic dependencies. Nix supports a
-similar but limited form of dynamic dependencies (contrary to what
-_Build Systems Ă la Carte_ says) through the `import` primitive of the
+similar but limited form of dynamic dependencies through
+the `import` primitive of the
Nix language, [which can take the result of a derivation
build](https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L74);
it does not attempt to gather the resulting `buildPaths` calls.
@@ -306,9 +332,11 @@ Another form of dynamic dependency is _derivation-building
derivations_
or _recursive derivations_, which were [recently implemented in
Nix](https://github.com/NixOS/nix/pull/3205). It supports another form
of dynamic dependency where the build process of a derivation can itself
-create and build derivations. It’s a great feature because in a
-nutshell, it allows Nix to be used not only to compose packages, but
-also at a finer grain as part of a package build process.
+create and build derivations (these are [_moldable
+tasks_](https://en.wikipedia.org/wiki/Parallel_task_scheduling_problem)
+in scheduling parlance). It’s a great feature because in a nutshell, it
+allows Nix to be used not only to compose packages, but also at a finer
+grain as part of a package build process.
Guix supports yet another form of dynamic dependencies. The newfangled
[`guix deploy`
@@ -326,10 +354,20 @@ along with all its dependencies on that target machine,
runs it, and
retrieves the result. This form of dynamic dependency also benefits
from the gathering machinery discussed above.
-# Conclusions
+# Conclusion
This is a long article on what may look like a fine point of Guix design
-and implementation, but there’s so much to say about it! Indeed, it
-took time to reach the current implementation. Grafts are key to the
-use of functional deployment in production because they enable for
-security updates.
+and implementation, but there’s much to say about it! Grafts are key to
+the use of functional deployment in production because they enable quick
+security updates, and it’s a lot better if they don’t harm the user
+experience.
+
+The pre-1.1.0 implementation of grafts had a negative impact on the user
+interface and on performance, which was due to the sequential handling
+of grafts, one package at a time. In 1.1.0 we addressed it by using
+delimited continuations to gather dynamic dependencies such as grafts,
+perform builds in bulk, and resume each derivation computation.
+
+As it turned out, the implementation of dynamic dependencies raises lots
+of interesting design and implementation issues, and it’s probably not
+the end of the road!
diff --git a/website/static/blog/img/inkscape-graft.dot
b/website/static/blog/img/inkscape-graft.dot
new file mode 100644
index 0000000..852ae15
--- /dev/null
+++ b/website/static/blog/img/inkscape-graft.dot
@@ -0,0 +1,17 @@
+digraph "Grafts" {
+ perl [ label = "perl", shape = box, fontname = Helvetica];
+ inkscape [ label = "inkscape", shape = box, fontname = Helvetica];
+ libc [ label = "glibc", shape = box, fontname = Helvetica ];
+
+ inkscape -> perl [ color = darkviolet ];
+ inkscape -> libc [ color = darkviolet ];
+ perl -> libc [ color = darkviolet ];
+
+ inkscapeg [ label = "inkscape (grafted)", shape = box, fontname = Helvetica,
+ color = dimgrey, fontcolor = dimgrey, margin = 0.2 ];
+ perlf [ label = "perl (fixed)", shape = box, fontname = Helvetica];
+
+ perlf -> libc [ color = darkviolet ];
+ inkscapeg -> inkscape [ color = orange ];
+ inkscapeg -> perlf [ color = orange ];
+}
\ No newline at end of file
diff --git a/website/static/blog/img/inkscape-graft.svg
b/website/static/blog/img/inkscape-graft.svg
new file mode 100644
index 0000000..3b6466c
--- /dev/null
+++ b/website/static/blog/img/inkscape-graft.svg
@@ -0,0 +1,79 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd";>
+<!-- Generated by graphviz version 2.40.1 (20161225.0304)
+ -->
+<!-- Title: Grafts Pages: 1 -->
+<svg width="198pt" height="267pt"
+ viewBox="0.00 0.00 198.00 267.00" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink";>
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 263)">
+<title>Grafts</title>
+<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-263 194,-263
194,4 -4,4"/>
+<!-- perl -->
+<g id="node1" class="node">
+<title>perl</title>
+<polygon fill="none" stroke="#000000" points="54,-108 0,-108 0,-72 54,-72
54,-108"/>
+<text text-anchor="middle" x="27" y="-86.3" font-family="Helvetica,sans-Serif"
font-size="14.00" fill="#000000">perl</text>
+</g>
+<!-- libc -->
+<g id="node3" class="node">
+<title>libc</title>
+<polygon fill="none" stroke="#000000" points="109,-36 55,-36 55,0 109,0
109,-36"/>
+<text text-anchor="middle" x="82" y="-14.3" font-family="Helvetica,sans-Serif"
font-size="14.00" fill="#000000">glibc</text>
+</g>
+<!-- perl->libc -->
+<g id="edge3" class="edge">
+<title>perl->libc</title>
+<path fill="none" stroke="#9400d3" d="M40.8788,-71.8314C47.2136,-63.5386
54.8384,-53.557 61.7926,-44.4533"/>
+<polygon fill="#9400d3" stroke="#9400d3" points="64.6452,-46.4847
67.9343,-36.4133 59.0825,-42.2353 64.6452,-46.4847"/>
+</g>
+<!-- inkscape -->
+<g id="node2" class="node">
+<title>inkscape</title>
+<polygon fill="none" stroke="#000000" points="117,-180 47,-180 47,-144
117,-144 117,-180"/>
+<text text-anchor="middle" x="82" y="-158.3"
font-family="Helvetica,sans-Serif" font-size="14.00"
fill="#000000">inkscape</text>
+</g>
+<!-- inkscape->perl -->
+<g id="edge1" class="edge">
+<title>inkscape->perl</title>
+<path fill="none" stroke="#9400d3" d="M68.1212,-143.8314C61.7864,-135.5386
54.1616,-125.557 47.2074,-116.4533"/>
+<polygon fill="#9400d3" stroke="#9400d3" points="49.9175,-114.2353
41.0657,-108.4133 44.3548,-118.4847 49.9175,-114.2353"/>
+</g>
+<!-- inkscape->libc -->
+<g id="edge2" class="edge">
+<title>inkscape->libc</title>
+<path fill="none" stroke="#9400d3" d="M82,-143.7623C82,-119.201 82,-75.2474
82,-46.3541"/>
+<polygon fill="#9400d3" stroke="#9400d3" points="85.5001,-46.0896 82,-36.0896
78.5001,-46.0897 85.5001,-46.0896"/>
+</g>
+<!-- inkscapeg -->
+<g id="node4" class="node">
+<title>inkscapeg</title>
+<polygon fill="none" stroke="#696969" points="181.5,-259 44.5,-259 44.5,-216
181.5,-216 181.5,-259"/>
+<text text-anchor="middle" x="113" y="-233.8"
font-family="Helvetica,sans-Serif" font-size="14.00" fill="#696969">inkscape
(grafted)</text>
+</g>
+<!-- inkscapeg->inkscape -->
+<g id="edge5" class="edge">
+<title>inkscapeg->inkscape</title>
+<path fill="none" stroke="#ffa500" d="M104.0335,-215.6623C100.7433,-207.6489
96.9722,-198.4647 93.4909,-189.986"/>
+<polygon fill="#ffa500" stroke="#ffa500" points="96.6195,-188.3906
89.5835,-180.4694 90.1441,-191.0494 96.6195,-188.3906"/>
+</g>
+<!-- perlf -->
+<g id="node5" class="node">
+<title>perlf</title>
+<polygon fill="none" stroke="#000000" points="190,-108 110,-108 110,-72
190,-72 190,-108"/>
+<text text-anchor="middle" x="150" y="-86.3"
font-family="Helvetica,sans-Serif" font-size="14.00" fill="#000000">perl
(fixed)</text>
+</g>
+<!-- inkscapeg->perlf -->
+<g id="edge6" class="edge">
+<title>inkscapeg->perlf</title>
+<path fill="none" stroke="#ffa500" d="M118.4595,-215.7359C124.9763,-189.7568
135.8855,-146.2671 143.0122,-117.8569"/>
+<polygon fill="#ffa500" stroke="#ffa500" points="146.4131,-118.6837
145.4515,-108.1326 139.6235,-116.9805 146.4131,-118.6837"/>
+</g>
+<!-- perlf->libc -->
+<g id="edge4" class="edge">
+<title>perlf->libc</title>
+<path fill="none" stroke="#9400d3" d="M132.8407,-71.8314C124.8489,-63.3694
115.1962,-53.1489 106.4584,-43.8971"/>
+<polygon fill="#9400d3" stroke="#9400d3" points="108.8011,-41.2802
99.3903,-36.4133 103.712,-46.0866 108.8011,-41.2802"/>
+</g>
+</g>
+</svg>
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- branch master updated: website: grafts-continued: Improve.,
Ludovic Courtčs <=