Re: [PATCH] tests/avocado: Disable the test_sbsaref_edk2_firmware in the gitlab-CI

[Thomas Huth]

On 10/07/2023 16.54, Peter Maydell wrote:
> On Mon, 10 Jul 2023 at 15:43, Thomas Huth <thuth@redhat.com> wrote:
> > The test fails occasionally, see e.g.:
> >
> >   https://gitlab.com/thuth/qemu/-/jobs/4196177756#L489
> >   https://gitlab.com/thuth/qemu/-/jobs/4623296271#L290
> >
> > Disable it at least in the gitlab-CI to avoid failing CI
> > pipelines due to this problem.
>
> I'm not really a fan of disabling flaky tests only in the
> gitlab CI. It means that plain "make check-avocado"
> is useless to the average developer because it can
> have all manner of broken or flaky tests in it that
> never get tested by our CI. In fact I've had to develop
> the habit of running "GITLAB_CI=1 make check-avocado"
> so I can run the CI tests that we think work and not
> get distracted by intermittent failures.
>
> If we want "disable this test except for if you're
> trying to investigate why it's flaky" then we should
> have the test be opt-in, not opt-out, the same way
> we do with QEMU_TEST_FLAKY_TESTS in the qtests.

Agreed, this makes perfectly sense. Actually, I was looking for something 
like this in the tests/avocado directory, but I did not find anything 
similar, so I went for the GITLAB_CI variable ...

I'll send a v2 with QEMU_TEST_FLAKY_TESTS instead (and document it in the 
docs directory).

 Thomas