bug#63412: Topological sorting in cuirass

[Ludovic Courtès]

Hi,

Andreas Enge <andreas@enge.fr> skribis:

> This is a wishlist bug, but it is important for architectures where we
> are currently short on build power, and where this issue can stall builds
> and waste an arbitrary amount of build power.
>
> Cuirass should sort builds and only offload derivations for which all
> inputs are available.

Cuirass has two build backends: talking to the build daemon, and using
the ZeroMQ-based “remote worker” protocol.  In practice we use the
latter, which fixes scalability issues with the former.

The worker protocol implements work stealing: workers periodically send
messages to “remote server” asking for work.  Said server replies with a
derivation for one of the systems the worker supports; that derivation
is chosen among the “builds” whose dependencies have all been
successfully built.

And here’s the trick: the server is doing the right thing, but it has a
partial view.  Namely, the server sees “builds” rather than
“derivations”.  “Builds” are the things explicitly declared in the
jobset.  If you have a declared build for GCC, but no build for MPC and
MPFR, then the server will consider that GCC has zero non-built
dependencies, even though MPFR and MPC may still need to be built.

Having ‘remote-worker’ operate on derivations rather than builds would
address this impedance mismatch, though there are complications (there
are bits of the database schema that amalgamate build/derivation).

Food for thought!

Ludo’.