Re: [Qemu-block] [Qemu-devel] [PATCH 2/6] tests/qemu-iotests/group: Introduce a new "ci" group for CI pipelines

[Thomas Huth]

On 24/04/2019 14.50, Daniel P. Berrangé wrote:
> On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote:
>> On 24/04/2019 13.25, Daniel P. Berrangé wrote:
>>> On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote:
>>>> Tests in this group are supposed to run in every possible QEMU 
>>>> configuration.
>>>> That means they should run with every QEMU binary (also non-x86), without
>>>> dependencies on an optional features, they must work at least with the 
>>>> qcow2
>>>> image format and be able to run on all kind of host filesystems and users
>>>> (i.e. also as "nobody" or "root").
>>>>
>>>> The initial list has been created as a subset of the "quick" group, where
>>>> I've disabled all tests that are failing with qemu-system-aarch64 or
>>>> qemu-system-tricore or in one of our CI pipelines.
>>>>
>>>> Signed-off-by: Thomas Huth <address@hidden>
>>>> ---
>>>>  tests/qemu-iotests/group | 194 ++++++++++++++++++++-------------------
>>>>  1 file changed, 102 insertions(+), 92 deletions(-)
>>>>
>>>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>>>> index bae7718380..2ed42dcc14 100644
>>>> --- a/tests/qemu-iotests/group
>>>> +++ b/tests/qemu-iotests/group
>>>> @@ -4,63 +4,73 @@
>>>>  # - do not start group names with a digit
>>>>  #
>>>>  
>>>> +#
>>>> +# Some notes about the groups:
>>>> +# - Tests in the "quick" group should finish within some few seconds
>>>> +# - Tests in the "ci" group are suitable for running in CI systems. That
>>>> +#   means they should run with every QEMU binary (also non-x86), with
>>>> +#   every QEMU configuration (i.e. no dependency on an optional feature),
>>>> +#   at least with the qcow2 image format and all kind of host filesystems
>>>> +#   and users (i.e. also as "nobody" or "root").
>>>
>>> No dependancy on optional features feels a bit restrictive from my POV.
>>>
>>> We have quite alot of testing coverage of stuff that depends on the
>>> crypto libraries that is important for us to run in "CI". This includes
>>> LUKS block drivers, qcow2 with LUKS,  NBD with TLS. Personally I expect
>>> all of those to be tested by CI systems.
>>>
>>> IOW for a "CI" group, the bar should be higher than "no optional features"
>>
>> Ok, I think I just did not write it properly. What I meant was: the test
>> should not *fail* because of a missing feature (what some tests are
>> unfortunately doing). It's fine if the test detects the missing feature
>> and then marks the test as *skip*.
> 
> Ok, this brings back the topic I don't think we ever resolved around how
> to best "detect" features in the iotests. eg if we want to know whether
> we can use TLS features, how do we decide this ?
> 
> One option is to do a dummy launch of QEMU using TLS & check if it fails,
> but that's kind of fragile for CI. A genuine bug in QEMU might cause this
> dummy launch to fail, and thus mistakenly skip all the tests instead of
> failing.

Maybe we need a "--dump-features" option for QEMU (or a QMP command)?
(somewhat similar to "gcc -dumpspecs" for example?)

 Thomas