bug#70475: updating my guix system linode server troubles

[Joshua Branson]

So my server that runs https://gnucode.me is a guix system linode server!  Woo
hoo!

Well lately the server is being a right pain, and it refuses to update.  Most
likely my server is lonely and just wants a hug.

#+BEGIN_SRC shell
joshua@gnucode.me $ guix system describe

Generation 20   Aug 17 2023 07:45:06    (current)
  file name: /var/guix/profiles/system-20-link
  canonical file name: /gnu/store/g10d07ab2hfspvj0wglwlicmphi9ly07-system
  label: GNU with Linux-Libre 6.3.13
  bootloader: grub
  root device: /dev/sda
  kernel: /gnu/store/224j20diq0qgxqzmh1hgsp35zi3qn9jc-linux-libre-6.3.13/bzImage
  channels:
    guix:
      repository URL: https://git.savannah.gnu.org/git/guix.git
      branch: master
      commit: 210bbf9c59355218fec1484a686cc2ecdc218506
    guixrus:
      repository URL: https://git.sr.ht/~whereiseveryone/guixrus
      branch: master
      commit: d70a9ea65eba43aba3ec7560d797452c01171b85
    nonguix:
      repository URL: https://gitlab.com/nonguix/nonguix
      branch: master
      commit: 1a3082c574077aaf6771ff927e725fc390a2d531
  configuration file: 
/gnu/store/8xi1rqijhlafnrvzrgla7k3qlkl9jsfw-configuration.scm

joshua@gnucode.me $ guix describe

Generation 2    Aug 17 2023 07:11:45    (current)
  guix 1b2d43f
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 1b2d43fe016848ea2ec16ff18cbc14340944fc4e

#+END_SRC

I have used guix deploy in the past to update my linode server, which works
super awesomely well! But lately, guix deploy no longer updates my linode
server.

#+BEGIN_SRC shell
joshua@laptopT400 $ guix deploy gnucode.me-current-config.scm -x -- sudo herd 
restart nginx
guix deploy: sending 12 store items (133 MiB) to '45.56.66.20'...
guix deploy: copertino: command succeeded
guix deploy: command output on copertino:
Service user-homes has been started.
Service nginx has been started.
#+END_SRC

Let's see if gnucode.me has been updated:

#+BEGIN_SRC shell
joshua@gnucode.me $ guix describe
                    Aug 2023  # (the same date as before)
joshua@gnucode.me $ guix system describe
                    Aug 2023  # (the same date as before)
#+END_SRC

/run/current-system/configuration.scm is the old configuration and not the new
one. What gives?

So it appears that guix deploy is not updating my server. That's annoying.  I've
been guix deploying my server happily for a while now, and I didn't even realize
that the server was failing to update.

Well, I had a similar issue, where my laptop was really outdated...guix pull was
failing. To update my laptop I built guix from source. I described that here:

https://issues.guix.gnu.org/68760

Well, since guix deploy fails to update my linode server, I tried running guix
pull: First I powered down the linode and updated the vm so that it had 4GB of
RAM.

Then at 2pm today, I ran guix pull.  It started building stuff, and I let it 
sit.
#+BEGIN_SRC shell
joshua@gnucode.me $ guix pull
  # more output above
building /gnu/store/lb5b7svdmfj1ijnzrripsjcv0bhqzpwb-Python-3.5.9.tar.xz.drv...
building 
/gnu/store/z8kqgsx6yh2mpm0n2jhzk1phihdscazf-datefudge_1.23.tar.xz.drv...
building /gnu/store/9cl9w041lwjz607f5l6pzjzgixdl1jjm-Python-3.5.9.tar.xz.drv...
building 
/gnu/store/8r73jwy621pvmbfyk34w0gqf65sff9di-elfutils-0.187.tar.bz2.drv...
building /gnu/store/0innk69jqi1zadpwyi8p9szdl95l2lk0-expat-2.5.0.tar.xz.drv...
building /gnu/store/3zysg963rlp01rpkjwsi8qzfr84kwd9d-gcc-11.3.0.tar.xz.drv...
building 
/gnu/store/lpkc0wsvgbfz0qwxn1nckh88n4xrg1kg-gettext-0.19.8.1.tar.gz.drv...
building /gnu/store/g6qb58plai65zdps620ny2rfif1z42s0-gettext-0.21.tar.gz.drv...
building /gnu/store/dc2y0rldz6wadknqz4jl6zjb7l4xsay5-glibc-2.35.tar.xz.drv...
building /gnu/store/gb6lfdzqprh1fmw91kpkclywg8sbrwg3-glibc-mesboot0-2.2.5.drv...
/ 'install' 
phase^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[|

building /gnu/store/5s4cfyc3w6jn958jn1i09g6my4g7l9l4-glibc-2.35.tar.xz.drv...
building /gnu/store/qm63mzpmzriksa9vi0gl977h7qhcmgkg-gcc-mesboot0-2.95.3.drv...
building /gnu/store/1hrskysmxkyblhw8mpkkz1m2jn4g0f8s-gmp-6.0.0a.tar.xz.drv...
building 
/gnu/store/cfbs6ysgyrqwfgbkd15gkbf7brgfgvk3-binutils-mesboot1-2.20.1a.drv...
building /gnu/store/b4xy58rqb3cqw2wiip8l0ibf9crk13zd-gmp-6.0.0a.tar.xz.drv...
building /gnu/store/1vp4cjnwxr55basnbjr4vfd5l5pmf41c-gnutls-3.8.3.tar.xz.drv...
building 
/gnu/store/j9vzm9yrkzfsgjpliab737fllq6kkm7q-guile-lib-0.2.8.tar.gz.drv...
building /gnu/store/wh7yqmqk07p7xby179razdbp6ysgqygl-isl-0.24.tar.xz.drv...
building /gnu/store/c3ds6y32xq9xck076apm525qqvp649ia-libffi-3.4.4.tar.gz.drv...
building 
/gnu/store/nn8hm6242hfwv906v1dy2m7sq3yv3hsh-libunistring-1.0.tar.xz.drv...
building /gnu/store/05nznjzl8qfsdps7wd51852babfvrndw-m4-1.4.19.tar.xz.drv...
building /gnu/store/6hqxh7hlcarn7l3y32pn4l3gkjbpkjia-make-mesboot-3.82.drv...
building 
/gnu/store/jxg62mw1619fznmnq3b7if5nv08s5q1i-module-import-compiled.drv...
building /gnu/store/gmw2a5zxc6liilr8jwpdw51isfrbrkis-gcc-mesboot1-4.6.4.drv...
- 'build' phas-
\
\ 'build' phas/^C

#+END_SRC

I started the guix pull command at 2pm today.  It's now 5pm, and the server is
trying to build gcc-mesboot0 or o

Let's check guix weather shall we?
#+BEGIN_SRC shell
joshua@gnucode.me $ guix weather
https://ci.guix.gnu.org ☀
  88.0% substitutes available (25,358 out of 28,805)
  at least 102,748.6 MiB of nars (compressed)
  177,798.9 MiB on disk (uncompressed)
  0.009 seconds per request (32.5 seconds in total)
  106.0 requests per second

  0.0% (0 out of 3,447) of the missing items are queued
  at least 1,000 queued builds
      aarch64-linux: 974 (97.4%)
      i686-linux: 14 (1.4%)
      x86_64-linux: 11 (1.1%)
      powerpc64le-linux: 1 (.1%)
  build rate: 22.87 builds per hour
      powerpc64le-linux: 2.10 builds per hour
      i686-linux: 10.75 builds per hour
      x86_64-linux: 9.42 builds per hour
      aarch64-linux: 0.53 builds per hour
      armhf-linux: 0.03 builds per hour
      i586-gnu: 0.05 builds per hour
looking for 28,805 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org ☀
  96.9% substitutes available (27,923 out of 28,805)
  at least 72,788.8 MiB of nars (compressed)
  216,904.7 MiB on disk (uncompressed)
  0.005 seconds per request (135.6 seconds in total)
  191.3 requests per second
  (continuous integration information unavailable)
#+END_SRC

Hmmm, that is really odd.

So let's try to update my linode server by building it from source eh? Since my
linode server only has 1 GB of RAM, I temporarily updated it to 4GB of RAM.


joshua@gnucode.me $ guix gc && guix shell --pure --check -D guix


That above command took 4 hours to complete. Guix shell took forever to build 
stuff. I saw some odd stuff that was getting built.  :(


./ bootstrap   -> worked



./configure   -> failed with:


checking for gzip... 
/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/gzip


checking for bzip2... 
/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/bzip2


checking for xz... /gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/xz


checking for git... no


configure: error: Git is missing; please install it.checking for bzip2... 
/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/bzip2


checking for gzip... 
/gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/gzip


checking for xz... /gnu/store/b914k2bjm3wixvd2kc9s03mzvfsif69c-profile/bin/xz


checking for git... no


configure: error: Git is missing; please install it.



I do have git installed on my server, but the "--pure" command blocked that.



So let's try doing what the manual actually says to do:



guix shell -D guix -CPW




./configure


checking for xz... /home/joshua/.guix-profile/bin/xz


checking for git... no


configure: error: Git is missing; please install it.



Ok, let's abide by the manual a little more and add in these helpful additional 
packages.


guix shell -D guix help2man git strace -CPW


./configure

make

sudo -E ./pre-inst-env guix system reconfigure 
../guix-config/linode-guix-system-configuration/gnucode.me-current-config.scm


The above command failed a few times, because I had to tweak the 
gnucode.me-current-config.scm file. First I had to change dovecot-service to 
dovecot-service-type. Then I had to get rid of the deploy stuff in the file. 
Then I was able to rerun the command and it worked:


sudo -E ./pre-inst-env guix system reconfigure 
../guix-config/linode-guix-system-configuration/gnucode.me-current-config.scm


This command also took a LONG time!  It was building glibc-mesboot0, and other 
very low level stuff.

Someone on irc said that I should take a look at what guix weather said

joshua@gnucode.me $ guix weather

guix weather: warning: using default substitute URLs
computing 30,045 package derivations for x86_64-linux...
looking for 35,187 store items on https://bordeaux.guix.gnu.org...
guix weather: warning: substitutes from 'https://bordeaux.guix.gnu.org' are 
unauthorized
hint: To authorize all substitutes from `https://bordeaux.guix.gnu.org' to be 
downloaded, the following command
needs to be run as root:

     guix archive --authorize <<EOF
     (public-key
      (ecc
       (curve Ed25519)
       (q #7D602902D3A2DBB83F8A0FB98602A754C5493B0B778C8D1DD4E0F41DE14DE34F#)
       )
      )

     EOF

Alternatively, on Guix System, you can add the signing key above to the 
`authorized-keys' field of
`guix-configuration'.

See "Getting Substitutes from Other Servers" in the manual for more information.

https://bordeaux.guix.gnu.org ☀
  94.8% substitutes available (33,354 out of 35,187)
  at least 74,939.1 MiB of nars (compressed)
  231,752.9 MiB on disk (uncompressed)
  0.016 seconds per request (573.7 seconds in total)
  61.3 requests per second
  (continuous integration information unavailable)
looking for 35,187 store items on https://ci.guix.gnu.org...
guix weather: warning: substitutes from 'https://ci.guix.gnu.org' are 
unauthorized
hint: To authorize all substitutes from `https://ci.guix.gnu.org' to be 
downloaded, the following command needs to
be run as root:

     guix archive --authorize <<EOF
     (public-key
      (ecc
       (curve Ed25519)
       (q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
       )
      )

     EOF

Alternatively, on Guix System, you can add the signing key above to the 
`authorized-keys' field of
`guix-configuration'.

See "Getting Substitutes from Other Servers" in the manual for more information.

https://ci.guix.gnu.org ☀
  98.0% substitutes available (34,499 out of 35,187)
  at least 158,714.5 MiB of nars (compressed)
  258,440.0 MiB on disk (uncompressed)
  0.005 seconds per request (179.5 seconds in total)
  196.0 requests per second

  0.0% (0 out of 688) of the missing items are queued
  at least 1,000 queued builds
      aarch64-linux: 998 (99.8%)
      i686-linux: 1 (.1%)
      powerpc64le-linux: 1 (.1%)
  build rate: 19.71 builds per hour
      x86_64-linux: 10.17 builds per hour
      aarch64-linux: 0.82 builds per hour
      powerpc64le-linux: 1.20 builds per hour
      i686-linux: 9.27 builds per hour
      armhf-linux: 0.06 builds per hour
      i586-gnu: 0.07 builds per hour

Ok let's try to authorize those gpg keys eh?

#+BEGIN_SRC shell
joshua@copertino ~/prog/gnu/guix/guix-src$ bash
joshua@copertino ~/prog/gnu/guix/guix-src$ su
Password:
root@copertino /home/joshua/prog/gnu/guix/guix-src# ./pre-inst-env      guix 
archive --authorize <<EOF
     (public-key
      (ecc
       (curve Ed25519)
       (q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
       )
      )

     EOF
>
>
>  # why is this not working?
> # I am pressing return...please be done
> # what more do you want me silly program?
> ^C
#+END_SRC

I'm pretty sure that didn't work... but let's check guix weather again to see if
my signing keys work.

#+BEGIN_SRC
./pre-inst-env guix weather

# I ran out of time and had to run
#+END_SRC

Hopefully I will update the server soon.  Any thoughts?

Thanks,

Joshua