[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Solstice infrastructure hackathon

From: Ludovic Courtès
Subject: Solstice infrastructure hackathon
Date: Thu, 16 Dec 2021 10:46:56 +0100
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)

Hello Guix!

This week, the node behind {ci,issues,disarchive} and was down twice for a few hours—nothing terrible in the end,
but it reminded us that, even though Guix doesn’t rely on any particular
machine, we can definitely feel the inconvenience when it’s down.

We were unlucky enough that it happened days after the other build farm,, ran out of disk space and had its CI stopped,
right before the big merge—so it doesn’t have substitutes for current

While discussing this on IRC the other day, we thought that perhaps it
was time to have an infrastructure hackathon.  How about Tuesday,
Dec. 21st?  (Probably with a followup in January.)

Here are tasks that were brought up:

  • Set up a backup server for, the head node of the, possibly moving some services such as the web site

  • Add DNS redundancy for so it can point to one of two
    hosts (need to figure out certbot challenges so both machines can
    update their certificates).

  • Set up with sysadmin status updates (possibly
    using Prometheus?).

  • Come up with a plan to add disks to the RAID array on bayfront, the
    head node of

  • Work on a plan to back up the Disarchive database currently on

  • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
    plain rsync or <>.

  • Have a documented procedure to set up substitute mirrors, such as
    the one in .cn (I can’t find the URL), ideally with plain rsync

Am I forgetting something?

Some of these tasks require root or physical access for the final steps,
but most of them are about (1) coming up with a plan, and (2) adjusting
the system configuration at
Anyone with sysadmin experience to share and Guix System knowledge can
join!  We’ll communicate over #guix on

Who’s in?  :-)


reply via email to

[Prev in Thread] Current Thread [Next in Thread]