Solstice infrastructure hackathon

[Ludovic Courtès]

Hello Guix!

This week, the node behind {ci,issues,disarchive}.guix.gnu.org and
guix.gnu.org was down twice for a few hours—nothing terrible in the end,
but it reminded us that, even though Guix doesn’t rely on any particular
machine, we can definitely feel the inconvenience when it’s down.

We were unlucky enough that it happened days after the other build farm,
bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped,
right before the big merge—so it doesn’t have substitutes for current
master.

While discussing this on IRC the other day, we thought that perhaps it
was time to have an infrastructure hackathon.  How about Tuesday,
Dec. 21st?  (Probably with a followup in January.)

Here are tasks that were brought up:

  • Set up a backup server for berlin.guix.gnu.org, the head node of the
    ci.guix.gnu.org, possibly moving some services such as the web site
    there.

  • Add DNS redundancy for guix.gnu.org so it can point to one of two
    hosts (need to figure out certbot challenges so both machines can
    update their certificates).

  • Set up status.guix.gnu.org with sysadmin status updates (possibly
    using Prometheus?).

  • Come up with a plan to add disks to the RAID array on bayfront, the
    head node of bordeaux.guix.gnu.org.

  • Work on a plan to back up the Disarchive database currently on
    berlin.guix.

  • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
    plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>.

  • Have a documented procedure to set up substitute mirrors, such as
    the one in .cn (I can’t find the URL), ideally with plain rsync
    access.

Am I forgetting something?

Some of these tasks require root or physical access for the final steps,
but most of them are about (1) coming up with a plan, and (2) adjusting
the system configuration at
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/>.
Anyone with sysadmin experience to share and Guix System knowledge can
join!  We’ll communicate over #guix on irc.libera.chat.

Who’s in?  :-)

Cheers,
Ludo’.