Prototype tool for building derivations

[Christopher Baines]

Hey,

Over the last couple of weeks, I've made some time to implement
something I was thinking about for a while.

In terms of getting to a point where Guix packages build reliably and
reproducibly, I think more testing is what's going to help. By taking
packages and building them more, on a wide variety of hardware and
software configurations, we'll get data on what works, what doesn't, and
where improvements and fixes can be made.

It's very much a prototype, but I've pushed some code up here [1] now,
the README.org file [2] contains usage instructions as well as a
description of the architecture.

1: https://git.cbaines.net/guix/build-coordinator/
2: https://git.cbaines.net/guix/build-coordinator/about/

So far, I've mostly done the boring stuff, but I'm excited about what
this could support.

Because the allocation/scheduling of builds is controlled, this offers
the possibility of doing some builds before others. If you were using
this for providing substitutes for example, it could be valuable to try
and prioritise building things that are requested more often, or those
that are more expensive (in time or space) to build.

Often there are concurrency issues with builds, I want to add a way of
specifying where builds should run. This would make it easy to test
building the same derivation in different setups, then capture where it
succeeds, fails, and how the output differs (if at all) across the
different environments.

I think it would be good to get point where there are many different
individuals and groups providing independent sources of Guix packages,
such that users can have a high level of confidence that the substitutes
they're getting correspond to the source code. Getting there will be
easier if substitute servers are easy to operate, and part of that I
think comes down to how easy it is to see what's going on. With the
current daemon implementation, I'm not sure how to get much data out
(this could be possible, I haven't looked very closely). This approach
however where the scheduling is done outside the daemon makes the
information more accessible.

I think some of the design decisions here are quite short sighted. I
think it would be better if some of this functionality could be handled
by the guix-daemon, especially things like providing information on
builds that are going to happen, but haven't happened yet. Once there's
a Guile implementation of the guix-daemon, hopefully some of this
technical debt can be repaid.

Just let me know if you have any questions or comments!

Thanks,

Chris

signature.asc
Description: PGP signature