bug#48468: substitute server connection timeout

[Ludovic Courtès]

Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

>> /gnu/store/qmzr030rzgikdxv3g9msqv0l8qp5j6y2-btrfs-raid-root-os.drv,
>> which was marked as failed earlier today due to missing .drv.  It’s a
>> 4KiB file, and the cache-bypass-threshold is ‘guix publish’ is typically
>> set to something much higher than that.  So ‘guix publish’ won’t return
>> 404 in that case.
>
> Yes but that derivation also depends on other derivations, for instance
> qemu-minimal and if I try:
>
> mathieu@berlin ~$ guix build qemu-minimal
> ...
> /gnu/store/lwv2pl0m6dkf6bkzip755w5p71g5akq4-qemu-minimal-7.1.0
>
>
> and then, from my machine.
>
> curl https://ci.guix.gnu.org/lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo
> We're baking it: /lwv2pl0m6dkf6bkzip755w5p71g5akq4.narinfo

Yes, but derivations (.drv) are depend only on “sources” (like
*-guile-builder, *.patch, *.scm) and on other derivations, all of which
are typically less than 1 MiB.

Derivations don’t depend on derivation outputs like that of qemu-minimal
above.

> So any build that requires a heavy substitute, heavier than the cache
> bypass threshold at least, will fail on the workers, as it would fail
> locally.

The main ‘guix publish’ instance on berlin has cache-bypass-threshold
set to 100 MiB; I don’t think we have any source or .drv that is this
big, or even a tenth of it.  :-)

The ‘guix publish’ instance spawned in (cuirass remote) doesn’t use
caching at all, so there’s no bypass threshold.

The only clue we have so far is a long chain of successful .drv
substitutions followed by a failing one:

--8<---------------cut here---------------start------------->8---
Downloading 
http://141.80.167.131/nar/zstd/9nm35401i8j42559iadi3iz3kmhmj7pr-guix-system-tests.drv...

[K guix-system-tests.drv  1KiB             0B/s 00:00 [                  ]   
0.0%
[K guix-system-tests.drv  1KiB         1.3MiB/s 00:00 [##################] 
100.0%
[K guix-system-tests.drv  1KiB         408KiB/s 00:00 [##################] 
100.0%

@ substituter-succeeded 
/gnu/store/9nm35401i8j42559iadi3iz3kmhmj7pr-guix-system-tests.drv
fetching path 
`/gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv'...
@ substituter-started 
/gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv 
substitute
Downloading 
http://141.80.167.131/nar/zstd/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv...

[K guix-system-tests-modules.drv  543B     0B/s 00:00 [                  ]   
0.0%
[K guix-system-tests-modules.drv  543B 851KiB/s 00:00 [##################] 
100.0%
[K guix-system-tests-modules.drv  543B 484KiB/s 00:00 [##################] 
100.0%

@ substituter-succeeded 
/gnu/store/9zf6inizcb09m136c44dj35fmzf2g3hs-guix-system-tests-modules.drv
cannot build missing derivation 
?/gnu/store/zq7idl0j51fdzqhhqm9ql90d0f2326k7-btrfs-root-on-subvolume-os.drv?
--8<---------------cut here---------------end--------------->8---

(From <https://ci.guix.gnu.org/build/224849/log/raw>.)

Plausible explanations that come to mind:

  1. ‘guix publish’ returning 404, but not due to baking.  Instead the
     .drv is simply not in store, hence 404.

  2. Client timeout (‘guix publish’ fails to reply on time).

  3. Cached 404 in nginx (though we’re not supposed to cache those I
     think?), or timeout in nginx (again due to ‘guix publish’ being too
     slow and ‘proxy_read_timeout’ is reached, currently 10s).

We should see if we can get useful info from nginx/publish logs.

Thanks,
Ludo’.