--- Begin Message ---
|
Subject: |
Poor 'guix substitute' performance when receiving Zstd-compressed substitutes |
|
Date: |
Tue, 22 Jun 2021 14:50:42 -0400 |
Hello,
It's something I've been observing for a while, but substitutes are very
IO intensive (as can be seen in iotop, the substitute process is waiting
on IO > 99% of the time) and is much slower than expected (3 minutes to
transfer 100 MiB uncompressed over a 50 mbps downstream link):
--8<---------------cut here---------------start------------->8---
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \
/gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix
substitute --substitute
--8<---------------cut here---------------end--------------->8---
The publisher (remote machine) is has its guix-daemon configured via:
--8<---------------cut here---------------start------------->8---
(service guix-publish-service-type
(guix-publish-configuration
(advertise? #t)
(compression '(("zstd" 3)))
(host "0.0.0.0"))) ;listen on all interfaces
--8<---------------cut here---------------end--------------->8---
CPU is idling during on both sides during the transfer (all the work
appears to be in the IO on the receiving end).
The above example was observed downloading
/gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I
do not think it is substitute-specific.
Maxim
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes |
|
Date: |
Thu, 06 Oct 2022 21:11:24 -0400 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) |
Hi,
Ludovic Courtès <ludo@gnu.org> writes:
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> It's something I've been observing for a while, but substitutes are very
>> IO intensive (as can be seen in iotop, the substitute process is waiting
>> on IO > 99% of the time) and is much slower than expected (3 minutes to
>> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>>
>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \
>> /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix
>> substitute --substitute
>>
>>
>> The publisher (remote machine) is has its guix-daemon configured via:
>>
>> (service guix-publish-service-type
>> (guix-publish-configuration
>> (advertise? #t)
>> (compression '(("zstd" 3)))
>> (host "0.0.0.0"))) ;listen on all interfaces
>
> Note that in this case nars are built and compressed on the fly on the
> server side, which puts an upper bound on the bandwidth you can achieve.
>
> I showed earlier how I profiled these things:
>
> https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/
>
> If the client is I/O-bound, that’s good: it means we can’t do any better
> (unless we skip unpacking as demonstrated by distri).
>
> If you can provide detailed profiles of either the server side or the
> client side (but in that case, make sure the server is caching things),
> that’d be great!
>
> Otherwise I’m afraid this is not actionable. :-)
Since moving from HDDs to SSDs, I haven't seen this problem, so I
suspect the poor IO of the HDDs was really the culprit rather than
something to do with guile-zstd (and we had also benchmarked the late
some when I experimented with using zstd-compressed man pages).
Closing!
--
Thanks,
Maxim
--- End Message ---