Re: Non-privileged daemons and offloading

[Ludovic Courtès]

Hello!

What you describe here is a hot topic and definitely a commonly
requested feature.  The difficulty here is that we’re hitting
limitations of the kernel, which requires root privileges to set up a
chroot and so on.

The way around it is Linux’ unprivileged “user namespaces”, as used by
‘guix environment --container’: they allow users to set up isolated
environments similar to what guix-daemon does, but without being root.
Unfortunately, this feature is disabled on some distros out of security
concerns (user namespaces are young and have a relatively bad track
record.)

You can check whether a system supports it like this:

  if [ -f /proc/self/ns/user ]
  then
      if [ -f /proc/sys/kernel/unprivileged_userns_clone ]
      then
          if [ `cat /proc/sys/kernel/unprivileged_userns_clone` -ne 0 ]
          then
              echo "unprivileged user namespaces supported"
          fi
      else
          echo "unprivileged user namespaces supported"
      fi
  fi

Regardless, it remains our best hope to support unprivileged daemons.

It would be nice to get stats on typical HPC systems.

Roel has been looking into these issues recently, so perhaps he has some
ideas.  The Nix daemon recently switch to user namespaces:

  https://github.com/NixOS/nix/commit/c68e5913c71badc89ff346d1c6948517ba720c93

We could backport this.  However, running builds with UID 0 is
potentially disruptive: some packages are sensitive to this and behave
differently under UID 0 (I remember Coreutils’ test suite does.)  Also,
this patch switches to user namespaces, but not specifically
_unprivileged_ user namespaces.

Using offloading as you suggest doesn’t help: you would still need a
daemon with access to /gnu/store.

(Thinking out loud.)

There’s a fun hack mind that could kinda work provided you use only
substitutes, where you wouldn’t even need a daemon:

  1. Compute the derivation of the package you want; normally that
     requires a daemon to which we make ‘add-to-store’ RPCs, but we
     should be able to fake them altogether;

  2. Use (guix scripts substitute) to download a substitute for that
     package, and unpack it under ~/.local, say;

  3. Use ‘call-with-container’ (thus, unprivileged user namespace) to
     put yourself in an environment where /gnu/store/foo inside is a
     bind-mount to ~/.local/gnu/store/foo outside.

There would remain the problem of profiles and grafts, which are normal
derivations.

When you think about it, it amounts to reimplementing (part of) the
daemon functionality as a library, which is probably the way to go.
That is, we could implement ‘add-to-store’ and ‘build-derivations’ such
that they would operate locally under ~/.local.  As a first milestone,
‘build-derivations’ could fail unless there’s a substitute available.

Food for thought!

It would probably help if one of us could work on it full time at some
point…

Thanks,
Ludo’.