Re: [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socket' qtest

[Daniel P . Berrangé]

Hi Jason,

As the net/ maintainer, could you take a look at this series.
This failure has been causing pain for CI for quite a while.

If you're happy with it, I can include it in a pending pull
request of other misc patches I have.

On Thu, Jan 04, 2024 at 04:29:36PM +0000, Daniel P. Berrangé wrote:
> We've previously bumped up the timeouts in the netdev-socket qtest
> to supposedly fix non-deterministic failures, however, the failures
> are still hitting CI.
> 
> A simple 'listen()' and 'connect()' pairing across 2 QEMU processes
> should be very quick to execute, even under high system load, so it
> was never likely that the test was failing due to timeouts being
> reached.
> 
> The actual root cause was a race condition in the test design. It
> was spawning a QEMU with a 'server' netdev, and then spawning one
> with the 'client' netdev. There was insufficient synchronization,
> however, so it was possible for the 2nd QEMU process to attempt
> to 'connect()' before the 'listen()' call was made by the 1st QEMU.
> 
> In the test scenarios that did not use the 'reconnect' flag, this
> would result in the client QEMU never getting into the expected
> state. The test code would thus loop on 'info network' until
> hitting the maximum wait time.
> 
> This series reverts the increased timeouts, and fixes synchronization
> in the test scenarios. It also improves reporting of errors in the
> socket netdev backend so that 'info network' reports what actually
> went wrong rather than a useless generic 'connection error' string.
> This will help us diagnose any future CI problems, should they occurr.
> 
> Daniel P. Berrangé (6):
>   Revert "netdev: set timeout depending on loadavg"
>   Revert "osdep: add getloadavg"
>   Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
>     seconds"
>   net: add explicit info about connecting/listening state
>   net: handle QIOTask completion to report useful error message
>   qtest: ensure netdev-socket tests have non-overlapping names
> 
>  include/qemu/osdep.h        | 10 ---------
>  meson.build                 |  1 -
>  net/stream.c                | 18 +++++++++++-----
>  tests/qtest/netdev-socket.c | 42 +++++++------------------------------
>  4 files changed, 21 insertions(+), 50 deletions(-)
> 
> -- 
> 2.43.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|