[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socke
From: |
Stefan Hajnoczi |
Subject: |
Re: [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socket' qtest |
Date: |
Thu, 4 Jan 2024 11:45:34 -0500 |
On Thu, 4 Jan 2024 at 11:30, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> We've previously bumped up the timeouts in the netdev-socket qtest
> to supposedly fix non-deterministic failures, however, the failures
> are still hitting CI.
>
> A simple 'listen()' and 'connect()' pairing across 2 QEMU processes
> should be very quick to execute, even under high system load, so it
> was never likely that the test was failing due to timeouts being
> reached.
>
> The actual root cause was a race condition in the test design. It
> was spawning a QEMU with a 'server' netdev, and then spawning one
> with the 'client' netdev. There was insufficient synchronization,
> however, so it was possible for the 2nd QEMU process to attempt
> to 'connect()' before the 'listen()' call was made by the 1st QEMU.
>
> In the test scenarios that did not use the 'reconnect' flag, this
> would result in the client QEMU never getting into the expected
> state. The test code would thus loop on 'info network' until
> hitting the maximum wait time.
>
> This series reverts the increased timeouts, and fixes synchronization
> in the test scenarios. It also improves reporting of errors in the
> socket netdev backend so that 'info network' reports what actually
> went wrong rather than a useless generic 'connection error' string.
> This will help us diagnose any future CI problems, should they occurr.
>
> Daniel P. Berrangé (6):
> Revert "netdev: set timeout depending on loadavg"
> Revert "osdep: add getloadavg"
> Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
> seconds"
> net: add explicit info about connecting/listening state
> net: handle QIOTask completion to report useful error message
> qtest: ensure netdev-socket tests have non-overlapping names
>
> include/qemu/osdep.h | 10 ---------
> meson.build | 1 -
> net/stream.c | 18 +++++++++++-----
> tests/qtest/netdev-socket.c | 42 +++++++------------------------------
> 4 files changed, 21 insertions(+), 50 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
- [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socket' qtest, Daniel P . Berrangé, 2024/01/04
- [PATCH 1/6] Revert "netdev: set timeout depending on loadavg", Daniel P . Berrangé, 2024/01/04
- [PATCH 2/6] Revert "osdep: add getloadavg", Daniel P . Berrangé, 2024/01/04
- [PATCH 3/6] Revert "tests/qtest/netdev-socket: Raise connection timeout to 120 seconds", Daniel P . Berrangé, 2024/01/04
- [PATCH 4/6] net: add explicit info about connecting/listening state, Daniel P . Berrangé, 2024/01/04
- [PATCH 6/6] qtest: ensure netdev-socket tests have non-overlapping names, Daniel P . Berrangé, 2024/01/04
- [PATCH 5/6] net: handle QIOTask completion to report useful error message, Daniel P . Berrangé, 2024/01/04
- Re: [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socket' qtest,
Stefan Hajnoczi <=
- Re: [PATCH 0/6] net: fix non-deterministic failures of the 'netdev-socket' qtest, Daniel P . Berrangé, 2024/01/09