[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [RFC 1/1] acceptance tests: rename acceptance to system
From: |
Willian Rampazzo |
Subject: |
Re: [RFC 1/1] acceptance tests: rename acceptance to system |
Date: |
Fri, 21 May 2021 14:49:02 -0300 |
On Fri, May 21, 2021 at 2:14 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 21/05/2021 16.29, Peter Maydell wrote:
> > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org>
> > wrote:
> >> If you think these tests belong to tests/tcg/, I am OK to put
> >> them they, but I don't think adding the Avocado buildsys
> >> machinery to the already-complex tests/tcg/ Makefiles is going
> >> to help us...
> >
> > This does raise the question of what we're actually trying
> > to distinguish. It seems to me somewhat that what tests/acceptance/
> > actually contains that makes it interestingly different from other
> > tests/ stuff is that it's specifically "tests using the Avocado
> > framework". On that theory we might name it tests/avocado/.
>
> I think there are two aspects:
>
> 1) These tests are using the avocado framework
>
> 2) These tests are downloading other stuff from the internet (unlike the
> other tests that we have)
>
After Peter's reply, I noticed QEMU does not organize tests under the
tests folder by software engineering test category but by the
mechanism/machinery the tests run on. This makes me think that we may
need to handle the folders name and the CI jobs name differently:
1 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-validation," and the GitLab CI job names to "validation,"
meaning that, in a promising future, other tests running on a
different framework and acting like validation tests would run in the
same make command and same GitLab CI job.
2 - Change the current "test/acceptance" folder name to "test/(avocado
or avoqado)." Change the "make check-acceptance" to "make
check-(avocado or avoqaco)" and the GitLab CI job names to "(avocado
or avoqado)," meaning that, in a promising future, we can categorize
validation jobs inside the CI and run each of the different validation
tests supported by a framework on its own GitLab CI job.
Personally, I prefer option 2 as it gives more flexibility to decide
how to set a GitLab CI job or run it when testing locally.
> > Or we could just leave it as it is -- is the current naming
> > actually confusing anybody? :-)
>
> Yes, I think "acceptance" is rather confusing. So far they haven't been part
> of your PR acceptance tests (well, now they are part of the gitlab-CI,
> though), and it's also not about tests that have been set up by customers,
> which is what you normally think of when hearing "acceptance tests". So a
> different name would be adequate.
>
> I think I'd vote for either "avocado", "avoqado" or "validation".
>
Even laughing every time I read "avoqado" (and thanks for that), I
liked the idea as there is supplementary code added inside
"tests/acceptance/avocado_qemu" to support the tests, meaning they are
not "pure" avocado.
> Thomas
>
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, (continued)
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Philippe Mathieu-Daudé, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Alex Bennée, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Philippe Mathieu-Daudé, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Peter Maydell, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Willian Rampazzo, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Willian Rampazzo, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Peter Maydell, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Willian Rampazzo, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Thomas Huth, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Philippe Mathieu-Daudé, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system,
Willian Rampazzo <=
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Alex Bennée, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Thomas Huth, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Philippe Mathieu-Daudé, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Alex Bennée, 2021/05/21
- Re: [RFC 1/1] acceptance tests: rename acceptance to system, Willian Rampazzo, 2021/05/21