Re: [Qemu-block] [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines

[Daniel P . Berrangé]

On Wed, Apr 24, 2019 at 01:50:46PM +0100, Daniel P. Berrangé wrote:
> On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote:
> > On 24/04/2019 13.25, Daniel P. Berrangé wrote:
> > > On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote:
> > >> Tests in this group are supposed to run in every possible QEMU 
> > >> configuration.
> > >> That means they should run with every QEMU binary (also non-x86), without
> > >> dependencies on an optional features, they must work at least with the 
> > >> qcow2
> > >> image format and be able to run on all kind of host filesystems and users
> > >> (i.e. also as "nobody" or "root").
> > >>
> > >> The initial list has been created as a subset of the "quick" group, where
> > >> I've disabled all tests that are failing with qemu-system-aarch64 or
> > >> qemu-system-tricore or in one of our CI pipelines.
> > >>
> > >> Signed-off-by: Thomas Huth <address@hidden>
> > >> ---
> > >>  tests/qemu-iotests/group | 194 ++++++++++++++++++++-------------------
> > >>  1 file changed, 102 insertions(+), 92 deletions(-)
> > >>
> > >> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> > >> index bae7718380..2ed42dcc14 100644
> > >> --- a/tests/qemu-iotests/group
> > >> +++ b/tests/qemu-iotests/group
> > >> @@ -4,63 +4,73 @@
> > >>  # - do not start group names with a digit
> > >>  #
> > >>  
> > >> +#
> > >> +# Some notes about the groups:
> > >> +# - Tests in the "quick" group should finish within some few seconds
> > >> +# - Tests in the "ci" group are suitable for running in CI systems. That
> > >> +#   means they should run with every QEMU binary (also non-x86), with
> > >> +#   every QEMU configuration (i.e. no dependency on an optional 
> > >> feature),
> > >> +#   at least with the qcow2 image format and all kind of host 
> > >> filesystems
> > >> +#   and users (i.e. also as "nobody" or "root").
> > > 
> > > No dependancy on optional features feels a bit restrictive from my POV.
> > > 
> > > We have quite alot of testing coverage of stuff that depends on the
> > > crypto libraries that is important for us to run in "CI". This includes
> > > LUKS block drivers, qcow2 with LUKS,  NBD with TLS. Personally I expect
> > > all of those to be tested by CI systems.
> > > 
> > > IOW for a "CI" group, the bar should be higher than "no optional features"
> > 
> > Ok, I think I just did not write it properly. What I meant was: the test
> > should not *fail* because of a missing feature (what some tests are
> > unfortunately doing). It's fine if the test detects the missing feature
> > and then marks the test as *skip*.
> 
> Ok, this brings back the topic I don't think we ever resolved around how
> to best "detect" features in the iotests. eg if we want to know whether
> we can use TLS features, how do we decide this ?
> 
> One option is to do a dummy launch of QEMU using TLS & check if it fails,
> but that's kind of fragile for CI. A genuine bug in QEMU might cause this
> dummy launch to fail, and thus mistakenly skip all the tests instead of
> failing.
> 
> We can't look at config-host.{h,mak} either as iotests are supposed to be
> runnable outside of a build tree.
> 
> We could look at whether gnutls is linked to QEMU but again that is going
> to be fragile if a bug in configure causes us to mistakenly stop enabling
> gnutls.
> 
> For developers, dynamically probing features & skipping is reasonable.
> 
> For automated CI, I think we should set clear desired featureset upfront
> based on what we know we are supposde to have available and not silently
> skip tests

I guess I should say "perfect is the enemy of good". What you propose
with "make check" is clearly better than what we have today which is
nothing by default.  So my point about defined featureset for running
under CI shouldn't be a blocker of the "make check" improvement.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|