[bug#39655] [PATCH core-updates] doc: Add 'Scheme-only Bootstrap' node.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello Ludo, Timothy!

Finally another version of the new bootstrap situation.  Timothy, please
have a look and feel free to change/add some things!

It's also getting time to write a blog post on this, would you like to
help out with that?

>> * doc/guix.texi (@detailmenu): Add Bootstrapping menu.
>> (Scheme-only Bootstrap): New node.
>> * doc/images/gcc-core-mesboot0-scheme-only-graph.dot: New image.
>> * doc/local.mk (DOT_FILES): Add it.
>
> Great!  I wonder if we need a new section though.  Do you think we could
> just keep it all in a single bootstrapping section that would describe
> the current state of affairs, possibly with a “Bootstrapping History”
> section or similar that would describe the milestones that got us there?
[..]
> That way, the main bootstrapping section would simply describe the
> current status in a standalone fashion.

Thanks for your thoughts on this.  I started to rework this by adding a
History section but in the end I removed it again, describing the
current situation while mentioning the gcc-triplet reduction in Guix
v1.1.

>> +@c
>> +packages@footnote{@c
>> +gash-boot,
>> +gash-utils-boot,
>> +%bootstrap-mes-rewired,
>> +gzip-mesboot,
>
> I wouldn’t list them all here because (1) it’s hard to understand what
> these names refer to, unless reading commencement.scm is your favorite
> hobby ;-), and (2) it’s bound to become outdated.
>
> WDYT?

Yes, instead of adding more of these, I also removed the previous list.

>> +As Gash and Gash Utils mature, and GNU packages become more

@Timothy: this is possibly touching very lightly on what Gash has
brought us; do you want to share some more here?

>> +bootstrappable (e.g., new releases of GNU Sed will also ship in
[..]
>> +@image{images/gcc-core-mesboot0-scheme-only-graph,6in,,Dependency graph of 
>> gcc-core-mesboot0}
>
> It would be nice to have a couple of lines commenting the graph, like:

Sure!  Updated version attached.

Greetings,
Janneke

>From b1341ecfc77345f1a423bbc66b6255a35613425e Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <address@hidden>
Date: Tue, 18 Feb 2020 11:36:37 +0100
Subject: [PATCH] doc: Update 'Bootstrapping' for further binary seed
 reduction.

* doc/images/gcc-core-mesboot0-graph.dot: New image, replacing
* doc/images/gcc-mesboot0-bag-graph.dot: ... remove file.
* doc/local.mk (DOT_FILES): Update for new image file.
* doc/guix.texi (Reduced Binary Seed Bootstrap): Use it in updated description
of further reduction of the trusted computing base.
---
 doc/guix.texi                          | 114 +++++++++++++------------
 doc/images/gcc-core-mesboot0-graph.dot | 114 +++++++++++++++++++++++++
 doc/local.mk                           |   2 +-
 3 files changed, 175 insertions(+), 55 deletions(-)
 create mode 100644 doc/images/gcc-core-mesboot0-graph.dot

diff --git a/doc/guix.texi b/doc/guix.texi
index 90324ce291..833509860c 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -35,7 +35,7 @@ Copyright @copyright{} 2016, 2017, 2018 Chris Marusich@*
 Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Efraim Flashner@*
 Copyright @copyright{} 2016 John Darrington@*
 Copyright @copyright{} 2016, 2017 Nikita Gillmann@*
-Copyright @copyright{} 2016, 2017, 2018, 2019 Jan Nieuwenhuizen@*
+Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Jan Nieuwenhuizen@*
 Copyright @copyright{} 2016 Julien Lepiller@*
 Copyright @copyright{} 2016 Alex ter Weele@*
 Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@*
@@ -28606,10 +28606,8 @@ Binutils, libc, and the other packages mentioned 
above---the
 @dfn{bootstrap binaries}.
 
 These bootstrap binaries are ``taken for granted'', though we can also
-re-create them if needed (more on that later).
-
-For @code{i686-linux} and @code{x86_64-linux} the Guix bootstrap process is
-more elaborate, @pxref{Reduced Binary Seed Bootstrap}.
+re-create them if needed (see @pxref{Preparing to Use the Bootstrap
+Binaries}).
 
 @menu
 * Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
@@ -28625,60 +28623,68 @@ Coreutils, Awk, Findutils, `sed', and `grep' and 
Guile, GCC, Binutils, and the
 GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap binaries are
 ``taken for granted.''
 
-Taking these binaries for granted means that we consider them to be a correct
-and trustworthy `seed' for building the complete system.  Therein lies a
-problem: the current combined size of these bootstrap binaries is about 250MB
-(@pxref{Bootstrappable Builds,,, mes, GNU Mes}).  Auditing or even inspecting
-these is next to impossible.
+Taking the bootstrap binaries for granted means that we consider them to
+be a correct and trustworthy `seed' for building the complete system.
+Therein lies a problem: the combined size of these bootstrap binaries is
+about 250MB (@pxref{Bootstrappable Builds,,, mes, GNU Mes}).  Auditing
+or even inspecting these is next to impossible.
 
-For @code{i686-linux} and @code{x86_64-linux}, Guix now features a ``Reduced
-Binary Seed'' bootstrap @footnote{We would like to say: ``Full Source
-Bootstrap'' and while we are working towards that goal it would be hyperbole
-to use that term for what we do now.}.
+For @code{i686-linux} and @code{x86_64-linux}, Guix now features a
+``Reduced Binary Seed'' bootstrap @footnote{We would like to say: ``Full
+Source Bootstrap'' and while we are working towards that goal it would
+be hyperbole to use that term for what we do now.}.
 
 The Reduced Binary Seed bootstrap removes the most critical tools---from a
 trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
 Library are replaced by: @code{bootstrap-mescc-tools} (a tiny assembler and
 linker) and @code{bootstrap-mes} (a small Scheme Interpreter and a C compiler
-written in Scheme and the Mes C Library, built for TinyCC and for GCC).  Using
-these new binary seeds and a new set of
-@c
-packages@footnote{@c
-nyacc-boot,
-mes-boot,
-tcc-boot0,
-tcc-boot,
-make-mesboot0,
-diffutils-mesboot,
-binutils-mesboot0,
-gcc-core-mesboot,
-mesboot-headers,
-glibc-mesboot0,
-gcc-mesboot0,
-binutils-mesboot,
-make-mesboot,
-gcc-mesboot1,
-gcc-mesboot1-wrapper,
-glibc-headers-mesboot,
-glibc-mesboot,
-gcc-mesboot,
-and
-gcc-mesboot-wrapper.
-}
-@c
-the ``missing'' Binutils, GCC, and the GNU C Library are built from source.
-From here on the more traditional bootstrap process resumes.  This approach
-has reduced the bootstrap binaries in size to about 130MB.  Work is ongoing to
-reduce this further.  If you are interested, join us on @code{#bootstrappable}
-on the Freenode IRC network.
-
-@c ./pre-inst-env guix graph --type=bag -e '(begin (use-modules (guix 
packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) 
gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot
-@c dot -T png doc/images/gcc-mesboot-bag-graph.dot > 
doc/images/gcc-mesboot-bag-graph.png
-
-Below is the generated dependency graph for @code{gcc-mesboot}, the bootstrap
-compiler used to build the rest of GuixSD.
-
-@image{images/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesboot}
+written in Scheme and the Mes C Library, built for TinyCC and for GCC).
+
+Using these new binary seeds the ``missing'' Binutils, GCC, and the GNU
+C Library are built from source.  From here on the more traditional
+bootstrap process resumes.  This approach has reduced the bootstrap
+binaries in size to about 145MB in Guix v1.1.
+
+The next step that Guix has taken is to replace the shell and all its
+utilities with implementations in Guile Scheme, the @emph{Scheme-only
+bootstrap}.  Gash (@pxref{Gash,,, gash, The Gash manual}) is a
+POSIX-compatible shell that replaces Bash, and it comes with Gash Utils
+which has minimalist replacements for Awk, the GNU Core Utilities, Grep,
+Gzip, Sed, and Tar.  The rest of the bootstrap binary seeds that were
+removed are now built from source.
+
+Building the GNU System from source is currently only possibly by adding
+some historical GNU packages as intermediate steps@footnote{Packages
+such as @code{gcc-2.95.3}, @code{binutils-2.14}, @code{glibc-2.2.5},
+@code{gzip-1.2.4}, @code{tar-1.22}, and some others for details see
+@file{gnu/packages/commencement.scm}}.  As Gash and Gash Utils mature,
+and GNU packages become more bootstrappable again (e.g., new releases of
+GNU Sed will also ship in @code{tar.gz} format), this set of added
+packages can hopefully be reduced again.
+
+The graph below shows the resulting dependency graph for
+@code{gcc-core-mesboot0}, the bootstrap compiler used for the
+traditional bootstrap of the rest of the Guix System.
+
+@c ./pre-inst-env guix graph -e '(@@ (gnu packages commencement) 
gcc-core-mesboot0)' | sed -re 
's,((bootstrap-mescc-tools|bootstrap-mes|guile-bootstrap).*shape =) box,\1 
ellipse,' > doc/images/gcc-core-mesboot0-graph.dot
+@image{images/gcc-core-mesboot0-graph,6in,,Dependency graph of 
gcc-core-mesboot0}
+
+The only significant binary bootstrap seeds that remain@footnote{
+Ignoring the 68KB @code{mescc-tools}; that will be removed later,
+together with @code{mes}.} are a Scheme intepreter and a Scheme
+compiler: GNU Mes and GNU Guile.
+
+This further reduction has brought down the size of the binary seed to
+about 60MB for @code{i686-linux} and @code{x86_64-linux}.
+
+Work is ongoing to remove all binary blobs from our free software
+bootstrap stack, working towards a Full Source Bootstrap.  Also ongoing
+is work to bring these bootstraps to the @code{arm-linux} and
+@code{aarch64-linux} architectures and to the Hurd.
+
+If you are interested, join us on @code{#bootstrappable} on the Freenode
+IRC network or discuss on @code{bug-mes@@gnu.org} or
+@code{gash-devel@@nongnu.org}.
 
 @node Preparing to Use the Bootstrap Binaries
 @section Preparing to Use the Bootstrap Binaries
@@ -28698,7 +28704,7 @@ guix graph -t derivation \
   | dot -Tps > gcc.ps
 @end example
 
-or, for the Reduced Binary Seed bootstrap
+or, for the further Reduced Binary Seed bootstrap
 
 @example
 guix graph -t derivation \
diff --git a/doc/images/gcc-core-mesboot0-graph.dot 
b/doc/images/gcc-core-mesboot0-graph.dot
new file mode 100644
index 0000000000..191582ed4c
--- /dev/null
+++ b/doc/images/gcc-core-mesboot0-graph.dot
@@ -0,0 +1,114 @@
+digraph "Guix package" {
+  "139781154519456" [label = "gcc-core-mesboot0@2.95.3", shape = box, fontname 
= Helvetica];
+  "139781154519456" -> "139781154519632" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520688" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971312" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520160" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971664" [color = darkgoldenrod];
+  "139781154519456" -> "139781154519984" [color = darkgoldenrod];
+  "139781154519456" -> "139781154519808" [color = darkgoldenrod];
+  "139781154519456" -> "139781154520512" [color = darkgoldenrod];
+  "139781154519456" -> "139781034971488" [color = darkgoldenrod];
+  "139781154519456" -> "139781034972896" [color = darkgoldenrod];
+  "139781154519456" -> "139781034972368" [color = darkgoldenrod];
+  "139781154519456" -> "139781034973072" [color = darkgoldenrod];
+  "139781154519456" -> "139781131267872" [color = darkgoldenrod];
+  "139781154519632" [label = "binutils-mesboot0@2.14", shape = box, fontname = 
Helvetica];
+  "139781154519632" -> "139781154520688" [color = peachpuff4];
+  "139781154519632" -> "139781034971312" [color = peachpuff4];
+  "139781154519632" -> "139781154520160" [color = peachpuff4];
+  "139781154519632" -> "139781034971664" [color = peachpuff4];
+  "139781154519632" -> "139781154519984" [color = peachpuff4];
+  "139781154519632" -> "139781154519808" [color = peachpuff4];
+  "139781154519632" -> "139781154520512" [color = peachpuff4];
+  "139781154519632" -> "139781034971488" [color = peachpuff4];
+  "139781154519632" -> "139781034972896" [color = peachpuff4];
+  "139781154519632" -> "139781034972368" [color = peachpuff4];
+  "139781154519632" -> "139781034973072" [color = peachpuff4];
+  "139781154519632" -> "139781131267872" [color = peachpuff4];
+  "139781154520688" [label = "bash-mesboot0@2.05b", shape = box, fontname = 
Helvetica];
+  "139781154520688" -> "139781034971488" [color = magenta];
+  "139781154520688" -> "139781034971840" [color = magenta];
+  "139781154520688" -> "139781034972896" [color = magenta];
+  "139781154520688" -> "139781034972368" [color = magenta];
+  "139781154520688" -> "139781034973072" [color = magenta];
+  "139781154520688" -> "139781131267872" [color = magenta];
+  "139781034971488" [label = "make-mesboot0@3.80", shape = box, fontname = 
Helvetica];
+  "139781034971488" -> "139781034971840" [color = dimgrey];
+  "139781034971488" -> "139781034972896" [color = dimgrey];
+  "139781034971488" -> "139781034972368" [color = dimgrey];
+  "139781034971488" -> "139781034973072" [color = dimgrey];
+  "139781034971488" -> "139781131267872" [color = dimgrey];
+  "139781034971840" [label = "tcc-boot0@0.9.26-1103-g6e62e0e", shape = box, 
fontname = Helvetica];
+  "139781034971840" -> "139781034972016" [color = magenta];
+  "139781034971840" -> "139781131266816" [color = magenta];
+  "139781034971840" -> "139781034972896" [color = magenta];
+  "139781034971840" -> "139781034972368" [color = magenta];
+  "139781034971840" -> "139781034973072" [color = magenta];
+  "139781034971840" -> "139781131267872" [color = magenta];
+  "139781034972016" [label = "mes-boot@0.22", shape = box, fontname = 
Helvetica];
+  "139781034972016" -> "139781034972192" [color = cyan3];
+  "139781034972016" -> "139781131266816" [color = cyan3];
+  "139781034972016" -> "139781034972896" [color = cyan3];
+  "139781034972016" -> "139781034972368" [color = cyan3];
+  "139781034972016" -> "139781034973072" [color = cyan3];
+  "139781034972016" -> "139781131267872" [color = cyan3];
+  "139781034972192" [label = "bootstrap-mes-rewired@0.19", shape = ellipse, 
fontname = Helvetica];
+  "139781034972192" -> "139781131266640" [color = darkseagreen];
+  "139781034972192" -> "139781034972896" [color = darkseagreen];
+  "139781131266640" [label = "bootstrap-mes@0", shape = ellipse, fontname = 
Helvetica];
+  "139781034972896" [label = "gash-boot@0.2.0", shape = box, fontname = 
Helvetica];
+  "139781034972896" -> "139781034973072" [color = darkviolet];
+  "139781034972896" -> "139781131267872" [color = darkviolet];
+  "139781034973072" [label = "bootar@1", shape = box, fontname = Helvetica];
+  "139781034973072" -> "139781131267872" [color = darkviolet];
+  "139781131267872" [label = "guile-bootstrap@2.0", shape = ellipse, fontname 
= Helvetica];
+  "139781131266816" [label = "bootstrap-mescc-tools@0.5.2", shape = ellipse, 
fontname = Helvetica];
+  "139781034972368" [label = "gash-utils-boot@0.1.0", shape = box, fontname = 
Helvetica];
+  "139781034972368" -> "139781034973072" [color = darkseagreen];
+  "139781034972368" -> "139781034972896" [color = darkseagreen];
+  "139781034972368" -> "139781131267872" [color = darkseagreen];
+  "139781034971312" [label = "bzip2-mesboot@1.0.8", shape = box, fontname = 
Helvetica];
+  "139781034971312" -> "139781034971488" [color = cyan3];
+  "139781034971312" -> "139781034971840" [color = cyan3];
+  "139781034971312" -> "139781034972896" [color = cyan3];
+  "139781034971312" -> "139781034972368" [color = cyan3];
+  "139781034971312" -> "139781034973072" [color = cyan3];
+  "139781034971312" -> "139781131267872" [color = cyan3];
+  "139781154520160" [label = "diffutils-mesboot@2.7", shape = box, fontname = 
Helvetica];
+  "139781154520160" -> "139781034971488" [color = red];
+  "139781154520160" -> "139781034971840" [color = red];
+  "139781154520160" -> "139781034972896" [color = red];
+  "139781154520160" -> "139781034972368" [color = red];
+  "139781154520160" -> "139781034973072" [color = red];
+  "139781154520160" -> "139781131267872" [color = red];
+  "139781034971664" [label = "gzip-mesboot@1.2.4", shape = box, fontname = 
Helvetica];
+  "139781034971664" -> "139781034971840" [color = red];
+  "139781034971664" -> "139781034972896" [color = red];
+  "139781034971664" -> "139781034972368" [color = red];
+  "139781034971664" -> "139781034973072" [color = red];
+  "139781034971664" -> "139781131267872" [color = red];
+  "139781154519984" [label = "patch-mesboot@2.5.9", shape = box, fontname = 
Helvetica];
+  "139781154519984" -> "139781034971488" [color = blue];
+  "139781154519984" -> "139781034971840" [color = blue];
+  "139781154519984" -> "139781034972896" [color = blue];
+  "139781154519984" -> "139781034972368" [color = blue];
+  "139781154519984" -> "139781034973072" [color = blue];
+  "139781154519984" -> "139781131267872" [color = blue];
+  "139781154519808" [label = "sed-mesboot0@1.18", shape = box, fontname = 
Helvetica];
+  "139781154519808" -> "139781034971488" [color = red];
+  "139781154519808" -> "139781034971840" [color = red];
+  "139781154519808" -> "139781034972896" [color = red];
+  "139781154519808" -> "139781034972368" [color = red];
+  "139781154519808" -> "139781034973072" [color = red];
+  "139781154519808" -> "139781131267872" [color = red];
+  "139781154520512" [label = "tcc-boot@0.9.27", shape = box, fontname = 
Helvetica];
+  "139781154520512" -> "139781034971312" [color = darkviolet];
+  "139781154520512" -> "139781034971488" [color = darkviolet];
+  "139781154520512" -> "139781034971840" [color = darkviolet];
+  "139781154520512" -> "139781034972896" [color = darkviolet];
+  "139781154520512" -> "139781034972368" [color = darkviolet];
+  "139781154520512" -> "139781034973072" [color = darkviolet];
+  "139781154520512" -> "139781131267872" [color = darkviolet];
+
+}
diff --git a/doc/local.mk b/doc/local.mk
index f8709bb3a4..aca1958edd 100644
--- a/doc/local.mk
+++ b/doc/local.mk
@@ -39,7 +39,7 @@ DOT_FILES =                                   \
   %D%/images/bootstrap-packages.dot            \
   %D%/images/coreutils-graph.dot               \
   %D%/images/coreutils-bag-graph.dot           \
-  %D%/images/gcc-mesboot-bag-graph.dot         \
+  %D%/images/gcc-core-mesboot0-graph.dot       \
   %D%/images/service-graph.dot                 \
   %D%/images/shepherd-graph.dot
 
-- 
2.26.0

-- 
Jan Nieuwenhuizen <address@hidden> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com