Re: [RFC 1/1] acceptance tests: rename acceptance to system

[Willian Rampazzo]

On Fri, May 21, 2021 at 4:16 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote:
> > On 5/20/21 9:53 PM, Willian Rampazzo wrote:
> >> Conceptually speaking, acceptance tests "are a series of specific tests
> >> conducted by the customer in an attempt to uncover product errors before
> >> accepting the software from the developer. Conducted by the end-user rather
> >> than software engineers, acceptance testing can range from an informal
> >> “test drive” to a planned and systematically executed series of scripted
> >> tests" [1]. Every time Pressman refers to the term "acceptance testing," he
> >> also refers to user's agreement in the final state of an implemented 
> >> feature.
> >> Today, QEMU is not implementing user acceptance tests as described by 
> >> Pressman.
> >>
> >> There are other three possible terms we could use to describe what is 
> >> currently
> >> QEMU "acceptance" tests:
> >>
> >>    1 - Integration tests:
> >>        - "Integration testing is a systematic technique for constructing 
> >> the
> >>           software architecture while at the same time conducting tests to
> >>           uncover errors associated with interfacing. The objective is to 
> >> take
> >>           unit-tested components and build a program structure that has 
> >> been
> >>           dictated by design." [2]
> >>        * Note: Sommerville does not have a clear definition of integration
> >>          testing. He refers to incremental integration of components inside
> >>          the system testing (see [3]).
>
> After thinking about this for a while, I agree with you that renaming the
> "acceptance" tests to "integration" tests is also not a good idea. When I
> hear "integration" test in the context of the virt stack, I'd rather expect
> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe
> virt-manager on top and tests them all together. So we should look for a
> different name indeed.
>
> >>    2 - Validation tests:
> >>        - "Validation testing begins at the culmination of integration 
> >> testing,
> >>           when individual components have been exercised, the software is
> >>           completely assembled as a package, and interfacing errors have 
> >> been
> >>           uncovered and corrected. At the validation or system level, the
> >>           distinction between different software categories disappears. 
> >> Testing
> >>           focuses on user-visible actions and user-recognizable output 
> >> from the
> >>           system." [4]
> >>        - "where you expect the system to perform correctly using a set of 
> >> test
> >>           cases that reflect the system’s expected use." [5]
> >>        * Note: the definition of "validation testing" from Sommerville 
> >> reflects
> >>          the same definition found around the Internet, as one of the 
> >> processes
> >>          inside the "Verification & Validation (V&V)." In this concept,
> >>          validation testing is a high-level definition that covers unit 
> >> testing,
> >>          functional testing, integration testing, system testing, and 
> >> acceptance
> >>          testing.
> >>
> >>    3 - System tests:
> >>        - "verifies that all elements mesh properly and that overall system
> >>           function and performance is achieved." [6]
> >>        - "involves integrating components to create a version of the 
> >> system and
> >>           then testing the integrated system. System testing checks that
> >>           components are compatible, interact correctly, and transfer the 
> >> right
> >>           data at the right time across their interfaces." [7]
> >>
> >> The tests implemented inside the QEMU "acceptance" directory depend on the
> >> software completely assembled and, sometimes, on other elements, like 
> >> operating
> >> system images. In this case, the proposal here is to rename the current
> >> "acceptance" directory to "system."
> >
> > Are user-mode tests using Avocado also system tests?
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html
>
> We've indeed got the problem that the word "system" is a little bit
> overloaded in the context of QEMU. We often talk about "system" when
> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user
> emulator binaries). For example, the "--disable-system" switch of the
> configure script, or the "build-system" and "check-system" jobs in the
> .gitlab-ci.yml file ... thus this could get quite confusing in the
> .gitlab-ci.yml file afterwards.

I agree with you here. After I made the changes to the code, I noticed
QEMU has the "system" word spread all over the place. That may confuse
people looking at the "system tests" without much interaction with
software testing terminology.

>
> So I think renaming "acceptance" to "system" is especially ok if we only
> keep the "softmmu"-related tests in that folder... would it maybe make sense
> to add the linux-user related tests in a separate folder called tests/user/
> instead, Philippe? And we should likely rename the current build-system and
> check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so?
>

As I mentioned in Philippe's reply, those tests are still considered
system tests because system testing is the software built and
interacting with external test artifacts in software engineering.

> Alternatively, what about renaming the "acceptance" tests to "validation"
> instead? That word does not have a duplicated definition in the context of
> QEMU yet, so I think it would be less confusing.

While at the beginning of your reply, I started thinking if
"validation" would cause less confusion for the QEMU project. Although
validation testing is a broader concept inside the Verification &
Validation process, encompassing unit testing, functional testing,
integration testing, system testing, and acceptance testing, it may be
an option for the QEMU project.

While system testing would be the correct terminology to use, if it
causes more confusion, using a less strict terminology, like
validation testing, is valid, in my opinion.

>
>   Thomas
>