Re: Guix on clusters and in HPC

[Ludovic Courtès]

Hi,

myglc2 <address@hidden> skribis:

> The scheduler that I am most familiar with, SGE, supports the
> proposition that compute hosts are heterogeneous and that they each have
> a fixed software and/or hardware configuration. As a result, users need
> to specify resources, such as SW packages &/or #CPUs &/or memory needed
> for a given job. These requirements in turn control where a given job
> can run. QMAKE, the integration of GNU Make with the SGE scheduler,
> further allows a make recipe step to specify specific resources for a
> SGE job to process the make step.

I see.

> While SGE is dated and can be a bear to use, it provides a useful
> yardstick for HPC/Cluster functionality. So it is useful to consider how
> Guix(SD) might impact this model. Presumably a defining characteristic
> of GuixSD clusters is that the software configuration of compute hosts
> no longer needs to be fixed and the user can "dial in" a specific SW
> configuration for each job step.  This is in many ways a good thing. But
> it also generates new requirements. How does one specify the SW config
> for a given job or recipe step:
>
> 1) VM image?
>
> 2) VM?
>
> 3) Installed System Packages?
>
> 4) Installed (user) packages?

The ultimate model here would be that of offloading¹: users would use
Guix on their machine, compute the derivation they want to build
locally, and offload the actual build to the cluster.  In turn, the
cluster would schedule builds on the available and matching compute
nodes.  But of course, this is quite sophisticated.

¹ https://www.gnu.org/software/guix/manual/html_node/Daemon-Offload-Setup.html

A more directly usable approach is to simply let users manage profiles
on the cluster using ‘guix package’ or ‘guix environment’.  Then they
can specify the right profile or the right ‘guix environment’ command in
their jobs.

> Based on my experiments with Guix/Debian, GuixSD, VMs, and VM images it
> is not obvious to me which of these levels of abstraction is
> appropriate. Perhaps any mix should be supported. In any case, tools to
> manage this aspect of a GuixSD cluster are needed. And they need to be
> integrated with the cluster scheduler to produce a manageable GuixSD HPC
> cluster.

Note that I’m focusing on the use of Guix on a cluster on top of
whatever ancient distro is already running, as a replacement for
home-made “modules” and such, and as opposed to running GuixSD on all
the compute nodes.

Running GuixSD on all the nodes of a cluster would certainly be valuable
from a sysadmin viewpoint, but it’s also something that’s much harder to
do in practice today.

> The most forward-thinking group that I know discarded their cluster
> hardware a year ago to replace it with starcluster
> (http://star.mit.edu/cluster/). Starcluster automates the creation,
> care, and feeding of a HPC clusters on AWS using the Grid Engine
> scheduler and AMIs. The group has a full-time "starcluster jockey" who
> manages their cluster and they seem quite happy with the approach. So
> you may want to consider starcluster as a model when you think of
> cluster management requirements.

Hmm, OK.

Thanks for your feedback,
Ludo’.