[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Mid-December update on bordeaux.guix.gnu.org
From: |
Ludovic Courtès |
Subject: |
Re: Mid-December update on bordeaux.guix.gnu.org |
Date: |
Mon, 20 Dec 2021 23:07:04 +0100 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) |
Hello!
Christopher Baines <mail@cbaines.net> skribis:
> In summary, the space issue I mentioned in the previous update has
> effectively been addressed. All the paused agents are now unpaused and
> builds are happening again.
Yay!
> However, due to the time spent not building things, the backlog is
> longer than usual, and the substitute availability (especially for
> x86_64-linux and i686-linux) is lower than usual.
Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now.
> ** Space issues and the nar-herder
>
> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
> bayfront was getting a little scarce. This week I started rolling out
> the nar-herder [2], a utility I've been thinking about for a while. This
> has enabled moving nars off of bayfront on to another machine which I've
> confusingly named lakefront.
Woow, neat!
Regarding nar-herder, I think it’d be nice to have a solution to
mirroring in Guix proper, developed similarly to other components,
because it could be a fairly central tool.
‘guix publish’ is probably not extensible enough to support that, but we
could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
> storage across 2 hard drives. When a nar is requested from bayfront, it
> will check it's local storage, and serve it from there if it exists,
> otherwise it will forward the request to lakefront. There might be a
> slight drop in the speed you can download nars, but apart from that this
> change shouldn't be visible.
Excellent, thanks for acting this promptly as problems crop up!
I think we should make sure this is funded by the project and that
credentials are shared. As discussed before, I think it’s best to keep
things tidy in that respect, in the spirit of building and maintaining
this collectively.
> The nar-herder is now busy deleting nars on bayfront which are available
> on lakefront. Once it's got through the backlog, I'll enable NGinx
> caching for the nars on bayfront, which should help improve
> performance. I've tested downloading the largest nars (~2GB) though, and
> it seems to work fine.
>
> In addition to lakefront, I've also added a 6TB hard drive to hatysa,
> the HoneyComb LX2 machine that I host. Like lakefront, it's busy
> downloading the nars from bayfront. This will act as a backup in case
> lakefront is lost.
>
> In general this is an important step in being more flexible where the
> nars are stored. There's still a reliance on storing pretty much all the
> nars on a single machine, but which machine has this role is more
> flexible now. I think this architecture also makes it easier to break
> the "all nars on a single machine" restriction in the future as well.
That’s really cool.
> Going forward, it would be good to have an additional full backup of the
> nars that bayfront can serve things from, to provide more
> redundancy. I'm hoping the nar-herder will also enable setting up
> geographically distributed mirrors, which will hopefully improve
> redundancy further, and maybe performance of fetching nars too.
>
> Once I've had a chance to neaten up the code a little, I'll get a Guix
> package and service written, plus I'll draft a Guix blog post about the
> nar-herder.
Usually I’m the one asking for blog posts :-), but I’d really like us as
a project to collectively engage on the topic before we publicize this
specific approach.
[...]
> That means there's the following currently running build agents (by
> architecture):
>
> - x86_64-linux + i686-linux (3 machines):
> - 4 core Intel NUC (giedi)
> - Max 16 cores for 1 concurrent build on bayfront
> - 32 cores on milano-guix-1 (slow storage though)
> - aarch64-linux + armhf-linux (2 machines)
> - 16 core HoneyComb LX2 (hatysa)
> - 4 core Overdrive (monokuma)
> - powerpc64le-linux (1 machine)
> - 64 core machine (polaris)
This is looking pretty nice! I’m also all for streamlining machine
handling, both administratively (in maintenance.git) and financially.
> Ironically, I think that the most under-resourced area is x86_64-linux +
> i686-linux. bayfront is restricted in what it can do since it also runs
> the coordinator, and things go badly if the machine gets
> overloaded. bayfront and milano-guix-1 both have hard drive storage,
> which also can slow them down when building things (especially
> milano-guix-1).
>
> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
> keep up, it would be good to make a plan to add capacity. I think even a
> single high core count x86_64-linux machine with performant storage
> would make a big difference.
Yes, we should do that, we still have funds for it.
> ** IPv6 not supported (yet)
>
> I was slow to notice, but bordeaux.guix.gnu.org isn't available over
> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
> to address this, but I haven't worked out quite how to yet.
Should be almost done now, as discussed on IRC today. \o/
Thanks!
Ludo’.
Re: Mid-December update on bordeaux.guix.gnu.org, Andreas Enge, 2021/12/17
Re: Mid-December update on bordeaux.guix.gnu.org,
Ludovic Courtès <=