Re: When substitute download + decompression is CPU-bound

[Ludovic Courtès]

Hi,

Guillaume Le Vaillant <glv@posteo.net> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> skribis:

[...]

>>> It’s not as nice as the ability to choose a download strategy, as we
>>> discussed earlier, but implementing that download strategy sounds
>>> tricky.
>>
>> If the user can choose their favourite substitute compression, I believe
>> it's usually enough since they are the best judge of their bandwidth /
>> hardware requirements.

As should be clear with what Guillaume and Nico posted, it’s pretty hard
to determine whether you need one compression algorithm or the other,
and it changes as you move your laptop around (different networking,
different CPU frequency scaling strategy, etc.).

> Here are a few numbers for the installation time in seconds (download
> time + decompression time) when fetching 580 MB of substitutes for
> download speeds between 0.5 MB/s and 20 MB/s.
>
> | Download speed | gzip -9 | lzip -9 | zstd -19 |
> |----------------+---------+---------+----------|
> |            0.5 |     287 |     151 |      181 |
> |            1.0 |     144 |      78 |       91 |
> |            1.5 |      97 |      54 |       61 |
> |            2.0 |      73 |      42 |       46 |
> |            2.5 |      59 |      35 |       37 |
> |            3.0 |      49 |      30 |       31 |
> |            3.5 |      42 |      27 |       26 |
> |            4.0 |      37 |      24 |       23 |
> |            4.5 |      33 |      22 |       21 |
> |            5.0 |      30 |      21 |       19 |
> |            5.5 |      28 |      19 |       17 |
> |            6.0 |      25 |      18 |       16 |
> |            6.5 |      24 |      17 |       14 |
> |            7.0 |      22 |      17 |       14 |
> |            7.5 |      21 |      16 |       13 |
> |            8.0 |      20 |      15 |       12 |
> |            8.5 |      18 |      15 |       11 |
> |            9.0 |      18 |      14 |       11 |
> |            9.5 |      17 |      14 |       10 |
> |           10.0 |      16 |      13 |       10 |
> |           11.0 |      15 |      13 |        9 |
> |           12.0 |      14 |      12 |        8 |
> |           13.0 |      13 |      12 |        8 |
> |           14.0 |      12 |      11 |        7 |
> |           15.0 |      11 |      11 |        7 |
> |           16.0 |      11 |      11 |        6 |
> |           17.0 |      10 |      10 |        6 |
> |           18.0 |      10 |      10 |        6 |
> |           19.0 |       9 |      10 |        5 |
> |           20.0 |       9 |      10 |        5 |
>
> When the download speed is lower than 3.5 MB/s, Lzip is better, and
> above that speed Zstd is better.
>
> As Gzip is never the best choice, it would make sense to drop it, even
> if we have to wait a little until everyone has updated their Guix daemon
> to a version with at least Lzip support.

Right.  We can drop it eventually, maybe soon since only 1% of our
downloads pick gzip.

> I think there are many people (like me) with a download speed slower
> than 3 MB/s, so like Pierre I would prefer keeping "lzip -9" and
> "zstd -19".

Understood.  To me, that means we need to implement something smart.

Ludo’.