Ludovic Courtès schreef op ma 28-02-2022 om 14:17 [+0100]:
> Hi,
>
> Maxime Devos <maximedevos@telenet.be> skribis:
>
> > 2. Instead of building all of Guix as a single derivation,
> > create a DAG of derivations. More concretely:
> >
> > First read the *.scm files to determine which module imports
> > which modules. Then to compile, say, (gnu packages acl),
> > a derivation taking gnu/packages/acl.scm and its dependencies
> > gnu/packages/attr.go, gnu/packages/base.go, ... is made
> > compiling gnu/packages/acl.scm to a gnu/packages/acl.go.
> >
> > Then to build all of Guix, 'union-build' or 'file-union' is used.
>
> This is what (guix self), used by ‘guix pull’, is already doing.
>
> However, currently, package modules are split in just two groups: the
> “base” group is the closure of (guix packages base), and the second
> group has all the rest:
>
> [...]
Looking at (guix self), it also has a few groups for non-package
modules (system tests, scripts, ...).
> At its core though, the situation pretty much reflects the free software
> situation: there are low-level packages (glibc, GCC, GTK, etc.) that
> might depend on high-level packages (Python, Pandoc, Rust, etc.).
>
> It’s not easy to split this spaghetti ball in smaller groups.
It's not easy to manually split the spaghetti, but we don't have
to, we could let the computer split the spaghetti for us (at least
partially, because of the circular imports), by computing the graph of
strongly-connected components and considering each SCC to be a ‘group’,
some of which depend on other groups, forming a DAG.
I was thinking about a bit of a different structure that can also be automated. My original idea was to use the already existing tree structure of the derivations, and split it based on depth. I think that gives a bit more structure, but might require splitting things that now are together (for example iirc sometimes we are defining bootstrap packages inheriting from the fully fledged ones, which introduces a syntactic dependency on something that is higher up the tree). Wdyt?
Regards,
g_bor
I believe Ricardo Wurmus has some script for computing the SCC?
Splitting large SCC in smaller parts can be left as an exercise
for later, it's a somewhat orthogonal concern.
> Thoughts?
I think it would be nice to let (guix self) automatically determine the
DAG of groups. It would reduce the ad-hocness of the *...-modules*
variables (some care required for patches, guix/man-db.scm, .js ...).
It would also make the node tree wide, which could reduce memory usage
(which might help with the ‘guix pull segfaults on i686-linux’
reports). In case of a crash (*), "guix pull" does not have to start
over from scratch, which would also help with those reports.
(*) This does not help with "failed to compute the derivation of Guix".
Some work would be required, but I think it will be worth it, and it
only has to be done once.
TBC, this was just an idea I wanted to share, I won't be working on it
in the forseeable future.
Greetings,
Maxime.