Re: guix weather -m etc/sources-manifest.scm and CI?

[zimoun]

Hi Chris,

On Fri, 17 Sep 2021 at 10:49, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> https://ci.guix.gnu.org
>>   74.6% substitutes available (12,556 out of 16,831)

[...]

>> https://bordeaux.guix.gnu.org
>>   99.8% substitutes available (16,804 out of 16,831)

>> The questions are:
>>
>> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
>> (I guess the missing ones on bordeaux are corner cases as icecat, 
>> linux-libre).
>
> bordeaux.guix.gnu.org is hopefully only missing substitutes for things
> where there's actually an issue building them. It's possible not to
> guess though and instead ask guix weather what is missing.

[...]

> If you visit https://data.guix.gnu.org and add the store path to the
> URL, you can get links to the failed builds ([1] for example).
>
> 1:
> https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz

Cool!  Thanks for explaining.  Let try to systematically address this
failures about the source. :-)


>> Does it make sense to duplicate the storage of all these origins?
>>
>> Using extensively "guix time-machine", I note that a lot of
>> derivations are missing and thus they are built locally, which is
>> costly on poor machine.  Could we reduce the duplication and so save
>> some space in order to systematically keep these derivations?  It
>> would greatly ease the Guix experience for "guix time-machine" users.
>> :-)
>
> In terms of duplication, I still think that an argument can be made that
> bordeaux.guix.gnu.org is providing value, even if it's doing some of the
> same stuff as ci.guix.gnu.org.

My question with duplication is about “long term storage of all the
sources”.  Not about building twice (it is easy to find justification
via reproducibility checker for instance :-)).

Now Guix have 2 build farms so let consider it is a strength. :-) The
question is how to exploit such strength by sharing the workload and
have a better global scaling up (and not 2 independent scaling).  For
what my opinion is worth here. :-)

> As for keeping build results, everything that's ever been built for
> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
> still around. In some ways, this is because deleting things is a bit
> more difficult, as the files aren't in the store, you can't just do a
> guix gc.
>
> However, I do want to try to never delete anything. That's going to
> require a bit of work as the local storage on the bayfront machine will
> be all used up at some point, but I do have the beginnings of a plan to
> avoid this an keep building things.

>From my understanding, the project does not have a lot of resources;
especially about storage.  And my question is: do we have a “coherent”
plan between the machines behind Bordeaux and the ones behind Berlin?

I mean, IMHO, it does not seem a correct usage of project resource to
back twice all from v1.0 to now (maybe from earlier to later).  And I
also have in mind energy consumption which is becoming more and more a
precious value.

For instance, we could imagine the both build farms build and challenge
the outputs.  Then, on Berlin the identical outputs are deleted after
say 6 months and only live on Bordeaux.  Bordeaux being the backup
time-machine.  That’s only an idea to open the discussion. :-) Not a
concrete plan.

Moreover, Disarchive is also something that should be added; probably to
machines behind Bordeaux in the above plan.

>From my point of view, what is really missing are the derivations for
*all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  And the backup time-machine should
systematically build them for all the reachable commits.


Cheers,
simon