Re: QEMU 5.1: Can we require each new device/machine to provided a test?

[John Snow]

On 5/19/20 5:04 AM, Daniel P. Berrangé wrote:
> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote:
>>
>>
>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote:
>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote:
>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote:
>>>>> Hello,
>>>>>
>>>>> Following Markus thread on deprecating unmaintained (untested) code
>>>>> (machines) [1] and the effort done to gather the information shared in
>>>>> the replies [2], and the various acceptance tests added, is it
>>>>> feasible to require for the next release that each new device/machine
>>>>> is provided a test covering it?
>>>>>
>>>>> If no, what is missing?
>>>>
>>>> If a qtest is feasible, yes, I think we should require one for new
>>>> devices. But what about machines - you normally need a test image for
>>>> this. In that case, there is still the question where testing images
>>>> could be hosted. Not every developer has a web space where they could
>>>> put their test images onto. And what about images that contain non-free
>>>> code?
>>>
>>> Yep, it isn't feasible to make this a hard rule.
>>>
>>> IMHO this is where a support tier classification comes into play
>>>
>>>  - Tier 1: actively maintained, qtest coverage available. Expected
>>>            to work reliably at all times since every commit is CI
>>>        tested
>>>
>>>   - Tier 2: actively maintained, no qtest coverage. Should usually
>>>            work but regression may creep in due to reliance on the
>>>        maintainer to manually test on adhoc basis
>>>
>>>   - Tier 3: not actively maintained, unknown state but liable to
>>>             be broken indefinitely
>>>
>>> Tier 1 is obviously the most desirable state we would like everthing to
>>> be at. Contributors will have to fix problems their patches cause as
>>> they will be blocked by CI.
>>>
>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in
>>> this tier will graduate to Tier 1 at some point. Even if it doesn't
>>> though, it is still valid to keep it in QEMU long term. Contributors
>>> shouldn't gratuitously break stuff in these board, but if they do,
>>> then the maintainer is ultimately responsible for fixing it, as the
>>> contributors don't have a test rig for it.
>>>
>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should
>>> not expect it to continue to exist long term. Contributors are free
>>> to send patches which break this, and are under no obligation to
>>> fix problems in these boards. We may deprecate & delete it after a
>>> while
>>>
>>>
>>> Over time we'll likely add more criteria to stuff in Tier 1. This
>>> could lead to some things dropping from Tier 1 to Tier 2. This is
>>> OK, as it doesn't make those things worse than they already were.
>>> We're just saying that Tier 2 isn't as thoroughly tested as we
>>> would like it to be in an ideal world.
>>
>> I really like the idea of device support tiers codified directly in the
>> QEMU codebase, to give upstream users some idea of which devices we
>> expect to work and which we ... don't, really.
>>
>> Not every last device we offer is enterprise production ready, but we
>> don't necessarily do a good job of explaining which devices fall into
>> which categories, and we've got quite a few of them.
>>
>> I wonder if a 2.5th tier would be useful; something like a "hobbyist"
>> tier for pet project SoC boards and the like -- they're not abandoned,
>> but we also don't expect them to work, exactly.
>>
>> Mild semantic difference from Tier 3.
> 
> I guess I was thinking such hobbyist stuff would fall into tier 2  if the
> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they
> largely aren't active on the mailing list responding to issues/questions.
> 
> We add have a 4 tier system overall and put hobbyist stuff at tier 3,
> and abandonware at tier 4.
> 
> Probably shouldn't go beyond 4 tiers though, as the more criteria we add
> the harder it is to clearly decide which tier something should go into.
> 
> The tier 1 vs 2 divison is clearly split based on CI which is a simple
> classification to decide on.
> 
> The tier 2 vs 3 division is moderately clearly split based on whether
> there is a frequently active maintainer.
> 
> We can probably squeeze in the 4th tier without too much ambiguity in
> the classisfication if we think it is adding something worthwhile either
> from our POV as maintainers, or for users consuming it.

Yes, I didn't mean to start watering it down into a 1,380 tier system
that we're never able to properly utilize.

I was thinking more along the lines of:

- Device works and is well loved
- Device works and is well loved (but we have to test manually)
- Device doesn't work, but is well loved
  (Use at your own peril, please file a bug report)
- Device doesn't work, and is unloved

Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where
things can shift from 1A to 1B as their test coverage allows, but it's
not meant to indicate general status otherwise.

Mostly, I would just like some way for users to avoid accidentally
running tier 2 or 3 devices /by accident/, or the ability to compile
QEMU versions that only allow tier 1 devices to be used.

It's all arbitrary -- but I think we agree more than not! I'd love to
have a list of first-class boards and devices that we promise to test
and have working.

--js