Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

[Philippe Mathieu-Daudé]

Cc'ing Markus

On 1/30/20 6:08 PM, Cornelia Huck wrote:
> On Thu, 30 Jan 2020 14:28:19 +0100
> Philippe Mathieu-Daudé <address@hidden> wrote:
>
> > On 1/30/20 1:26 PM, Cornelia Huck wrote:
> > > On Wed, 29 Jan 2020 22:23:20 +0100
> > > Philippe Mathieu-Daudé <address@hidden> wrote:
> > >    
> > > > Introduce the 'version-min' tag.
> > > > Tests can set it to the minimum version of QEMU they require.
> > > > If QEMU is older, the tests will be cancelled (skipped):
> > > >
> > > >     $ python -m avocado --show=app run 
> > > > tests/acceptance/x86_cpu_model_versions.py
> > > >      (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test 
> > > > expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > >      (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version 
> > > > '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > >      (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version 
> > > > '4.1' but QEMU binary is '3.1.1' (0.10 s)
> > > >      ...
> > > >
> > > > Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
> > > > ---
> > > >    tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
> > > >    1 file changed, 18 insertions(+)
> > >
> > > Hm... this might cause headaches if someone tries to backport tests on
> > > distro versions; but we probably won't be able to test for the feature
> > > in many cases anyway?
> > >    
> >
> > Are you saying a distro version might not match upstream version when a
> > feature is introduced? Isn't it the point of versioning? If you are
> > thinking of some case you point me at it so I can improve this patch?
>
> Something like the following:
>
> - a feature together with a test is introduced in QEMU version n
> - a distro is using m (where m < n), but wants the new feature and
>    backports it
>
> If that distro now wants to include the acceptance test as well (to run
> in their internal testing), they need to adapt the version check as
> well, or the test simply won't run. Depending on how this is reported,
> it might not be easily noticed. If the test depends on another feature
> (but the feature does not), it might get a bit more complicated.
>
> It's probably not a big problem, but I wanted to point it out.

OK now I understand better.

I wrote that patch having QAPI versioned features in mind, which are 
versioned/documented.
For example:

  ##
  # @qom-list-types:
  #
  ...
  #
  # Since: 1.1

Now it is true that we add a new feature to a device, we do not document 
that in QAPI, and I don't see how we can test it via QMP.