Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Philippe Mathieu-Daudé]

On 5/21/21 7:14 PM, Thomas Huth wrote:
> On 21/05/2021 16.29, Peter Maydell wrote:
>> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé <f4bug@amsat.org>
>> wrote:
>>> If you think these tests belong to tests/tcg/, I am OK to put
>>> them they, but I don't think adding the Avocado buildsys
>>> machinery to the already-complex tests/tcg/ Makefiles is going
>>> to help us...
>>
>> This does raise the question of what we're actually trying
>> to distinguish. It seems to me somewhat that what tests/acceptance/
>> actually contains that makes it interestingly different from other
>> tests/ stuff is that it's specifically "tests using the Avocado
>> framework". On that theory we might name it tests/avocado/.
> 
> I think there are two aspects:
> 
> 1) These tests are using the avocado framework
> 
> 2) These tests are downloading other stuff from the internet (unlike the
> other tests that we have)
> 
>> Or we could just leave it as it is -- is the current naming
>> actually confusing anybody? :-)
> 
> Yes, I think "acceptance" is rather confusing. So far they haven't been
> part of your PR acceptance tests (well, now they are part of the
> gitlab-CI, though), and it's also not about tests that have been set up
> by customers, which is what you normally think of when hearing
> "acceptance tests". So a different name would be adequate.

IIUC the current "acceptance tests" are the ones Peter runs,
which are *gating* the merge process. They can not fail.

The current tests in tests/acceptance/ use Avocado (as said Thomas,
to easily download artifacts) and shouldn't be considered gating;
they could fail.

This is my confusion so far. It should be OK to add tests using the
Avocado framework which might fail and aren't gating.