bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes

[Maxim Cournoyer]

Hello,

It's something I've been observing for a while, but substitutes are very
IO intensive (as can be seen in iotop, the substitute process is waiting
on IO > 99% of the time) and is much slower than expected (3 minutes to
transfer 100 MiB uncompressed over a 50 mbps downstream link):

--8<---------------cut here---------------start------------->8---
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
13934 be/4 root     1033.09 K/s 1485.06 K/s  0.00 % 93.36 % guile \ 
/gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix 
substitute --substitute
--8<---------------cut here---------------end--------------->8---

The publisher (remote machine) is has its guix-daemon configured via:

--8<---------------cut here---------------start------------->8---
      (service guix-publish-service-type
               (guix-publish-configuration
                (advertise? #t)
                (compression '(("zstd" 3)))
                (host "0.0.0.0")))      ;listen on all interfaces
--8<---------------cut here---------------end--------------->8---

CPU is idling during on both sides during the transfer (all the work
appears to be in the IO on the receiving end).

The above example was observed downloading
/gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I
do not think it is substitute-specific.

Maxim