Re: Mid-December update on bordeaux.guix.gnu.org

[Christopher Baines]

Ludovic Courtès <ludo@gnu.org> writes:

>> However, due to the time spent not building things, the backlog is
>> longer than usual, and the substitute availability (especially for
>> x86_64-linux and i686-linux) is lower than usual.
>
> Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now.

I've been so slow in replying to this email that the situation has
actually improved now, substitute availability for x86_64-linux is
around ~70% and still slowly rising.

>> ** Space issues and the nar-herder
>>
>> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
>> bayfront was getting a little scarce. This week I started rolling out
>> the nar-herder [2], a utility I've been thinking about for a while. This
>> has enabled moving nars off of bayfront on to another machine which I've
>> confusingly named lakefront.
>
> Woow, neat!
>
> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.
>
> ‘guix publish’ is probably not extensible enough to support that, but we
> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.

I think having something in Guix would be good too.

I still view the nar-herder as a bit wierd in terms of the collection of
concerns, mixing mirroring in with moving nars around between machines
and I'm hoping to add some metrics functionality soon as well.

>> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
>> storage across 2 hard drives. When a nar is requested from bayfront, it
>> will check it's local storage, and serve it from there if it exists,
>> otherwise it will forward the request to lakefront. There might be a
>> slight drop in the speed you can download nars, but apart from that this
>> change shouldn't be visible.
>
> Excellent, thanks for acting this promptly as problems crop up!
>
> I think we should make sure this is funded by the project and that
> credentials are shared.  As discussed before, I think it’s best to keep
> things tidy in that respect, in the spirit of building and maintaining
> this collectively.

That would be good, I can start a thread on guix-sysadmin about this.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>>
>> Once I've had a chance to neaten up the code a little, I'll get a Guix
>> package and service written, plus I'll draft a Guix blog post about the
>> nar-herder.
>
> Usually I’m the one asking for blog posts :-), but I’d really like us as
> a project to collectively engage on the topic before we publicize this
> specific approach.

Sure, I also still need to post patches for the Guix service, which I'll
try to do soon.

>> That means there's the following currently running build agents (by
>> architecture):
>>
>>  - x86_64-linux + i686-linux (3 machines):
>>    - 4 core Intel NUC (giedi)
>>    - Max 16 cores for 1 concurrent build on bayfront
>>    - 32 cores on milano-guix-1 (slow storage though)
>>  - aarch64-linux + armhf-linux (2 machines)
>>    - 16 core HoneyComb LX2 (hatysa)
>>    - 4 core Overdrive (monokuma)
>>  - powerpc64le-linux (1 machine)
>>    - 64 core machine (polaris)
>
> This is looking pretty nice!  I’m also all for streamlining machine
> handling, both administratively (in maintenance.git) and financially.

I think there was some discussion previously about guix-europe buying
the HoneyComb board, which I can probably restart. It would be good to
also sort out better hosting for the powerpc64le-linux machine.

I'll try to put some time in to organising things in maintenance.git.

>> Ironically, I think that the most under-resourced area is x86_64-linux +
>> i686-linux. bayfront is restricted in what it can do since it also runs
>> the coordinator, and things go badly if the machine gets
>> overloaded. bayfront and milano-guix-1 both have hard drive storage,
>> which also can slow them down when building things (especially
>> milano-guix-1).
>>
>> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
>> keep up, it would be good to make a plan to add capacity. I think even a
>> single high core count x86_64-linux machine with performant storage
>> would make a big difference.
>
> Yes, we should do that, we still have funds for it.

Great :)

>> ** IPv6 not supported (yet)
>>
>> I was slow to notice, but bordeaux.guix.gnu.org isn't available over
>> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
>> to address this, but I haven't worked out quite how to yet.
>
> Should be almost done now, as discussed on IRC today.  \o/

Indeed, I think this is working now.

Thanks,

Chris

signature.asc
Description: PGP signature