Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo, zimoun, all,

Thanks again for your answers, for the links to documentation, and 
for the support on IRC!

> Hmm could it be that nscd is not running, and that 
> /etc/nsswitch.conf
> specifies a “non-standard” NSS plugin, such as ‘sssd’?
>
>   
> https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch-1
>
> Does ‘guix install hello --no-offload’ work around the issue?

Yes, that solves the issue!

> >  $ guix archive --authorize < \
> >    
> > /gnu/store/xb4szjambyi52bpnkjv080g2mlfqqpp0-profile/share/guix/ci.guix.gnu.org.pub
> >  guix archive: error: mkdir: Permission denied
>
> I guess it’s trying to write to /etc/guix.  You probably need to 
> set
> GUIX_STATE_DIRECTORY=$HOME/.local/var/guix here.

Yes, that would be GUIX_CONFIGURATION_DIRECTORY. As discussed on 
IRC though, /etc in the tarball is a symlink to the profile, so 
neither `guix archive` nor I can create /etc/guix. So I resorted 
to creating a `/etc-guix` folder which I use for 
GUIX_CONFIGURATION_DIRECTORY.

> > 2. knowing that my environment has user namespaces (and knowing 
> > that
> > that may not be enough), would it be possible to set up 
> > guix-daemon as
> > non-root with the assistance of a sysadmin (so say with root 
> > access
> > during setup), with build users and all, and have it provide 
> > identical
> > guarantees to a guix-daemon running as root? If not, why not, 
> > and
> > could we work around that?
>
> Good question.  It should be possible to make the daemon run as
> non-root; that’s what the trick with the ‘guix pack -R’ wrapper 
> should
> achieve, but it could also be a built-in capability.

Yes, I'd be very happy to try and build that once things work!

So, to summarize the current status, here is what I do:

On my Guix System (`deigo` is my ssh config for the cluster):

 $ scp (guix pack -R -S /bin=bin -S /etc=etc guix bash) 
 deigo:guix-pack.tar.gz

Then on deigo:

 $ mkdir -p /a-working-directory/guix
 $ cd /a-working-directory/guix
 $ tar xzf ~/guix-pack.tar.gz
 $ mkdir etc-guix
 $ bin/bash
 $ . etc/profile
 $ export GUIX_STATE_DIRECTORY=$(pwd)/var/guix
 $ export GUIX_LOG_DIRECTORY=$(pwd)/var/log
 $ export GUIX_CONFIGURATION_DIRECTORY=$(pwd)/etc-guix

Then the fun starts. There are 4 different cases and problems, 
depending on whether I use substitutes or not, and on whether 
`/a-working-directory` is an NFS share or a local mount.


Case 1: with substitutes, on a local (non-NFS) folder
-----------------------------------------------------

 $ guix archive --authorize < 
 bin/../share/guix/ci.guix.gnu.org.pub
 $ guix-daemon --disable-chroot &
 $ guix install hello
 [... dowloading ...]
 unexpected substituter message 'defining `GUIX_LOCPATH', along 
 these lines:

      guix install glibc-utf8-locales
      export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

 See the "Application Setup" section in the manual, for more 
 info.
 '

I think this is https://issues.guix.gnu.org/45166.

Using instead `guix pack -R -S /bin=bin -S /etc=etc -S /lib=lib 
guix bash glibc-utf8-locales`, and:

 $ export GUIX_LOCPATH=$(pwd)/lib/locale
 $ ls $GUIX_LOCPATH/
 2.31

seems to start working, but then `guix install hello` fails with 
the same error, and:

 $ ls $GUIX_LOCPATH/
 ls: cannot access '/tmp/sebastien-lerique/guix/lib/locale/': No 
 such file or directory

Is it possible that glibc-utf8-locales gets garbage-collected?


Case 2: from source (no substitutes), on a local (non-NFS) folder
-----------------------------------------------------------------

No need for the locale fix here (another bug appears before that):

 $ guix-daemon --disable-chroot &
 # --no-offload works around the missing nscd problem; up to now 
 this
 # doesn't seem necessary when using substitutes
 $ guix install hello --no-offload
 [... builds for a while ...]
 build of 
 /gnu/store/pkn1w1q3xkn273kpmggc4dnq6n6hr9jy-bzip2-mesboot-1.0.8.drv 
 failed

The build log for bzip2-mesboot ends with:

 tcc -ar cq libbz2.a blocksort.o huffman.o crctable.o randtable.o 
 compress.o decompress.o bzlib.o
 ranlib libbz2.a
 /gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0/bin/sh: 
 ranlib: Command not found.
 make: *** [libbz2.a] Error 127
 command "make" "CC=tcc -I ." "AR=tcc -ar" "bzip2" 
 "PREFIX=/gnu/store/s94hyrv1vgllxir5niiyzfc9g80l5kcd-bzip2-mesboot-1.0.8" 
 failed with status 2

Is this new and should be reported as a bug? Should 
binutils-mesboot0 be part of bzip2-mesboot's inputs? (I haven't 
learned about the boostrapping mechanism yet.)


Case 3: with substitutes, on an NFS share
-----------------------------------------

Again no need for the locale fix (another bug appears before 
that):

 $ guix archive --authorize < 
 bin/../share/guix/ci.guix.gnu.org.pub
 $ guix-daemon --disable-chroot &
 $ guix install hello
 [... downloading ...]
 guix install: error: cannot unlink 
 `/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib': 
 Directory not empty

so checking:

 $ ls -lA 
 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib
 total 3840
 -r-xr-xr-x 1 sebastien-lerique froeseuni   64832 Jan  1  1970 
 .nfs000000000172caa5000054bd
 [... more .nfs files ...]

 $ lsof -f -- 
 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd` 
 reports:
 [... some warnings ...]
 COMMAND     PID              USER  FD   TYPE DEVICE SIZE/OFF 
 NODE NAME
 bash      25616 sebastien-lerique mem    REG   0,52    64832 
 24300197 
 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd
 guix-daem 25934 sebastien-lerique mem    REG   0,52    64832 
 24300197 
 /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd

Not sure what's going wrong here. Maybe it's more 
garbage-collecting (as in case 1), failing because of an nfs 
nuance or bug?


Case 4: from source (no substitutes), on an NFS share
-----------------------------------------------------

Fails in the exact same way as Case 2.


I have no idea how to make progress in any of these cases (except 
the bzip2-mesboot one maybe), so any input on these different 
problems is very welcome :)

Thanks and happy hacking!
Sébastien