Re: QEMU 5.1: Can we require each new device/machine to provided a test?

[Daniel P . Berrangé]

On Tue, May 19, 2020 at 07:06:40PM -0400, John Snow wrote:
> 
> 
> On 5/19/20 5:04 AM, Daniel P. Berrangé wrote:
> > On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote:
> >>
> >>
> >> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote:
> >>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote:
> >>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote:
> >>>>> Hello,
> >>>>>
> >>>>> Following Markus thread on deprecating unmaintained (untested) code
> >>>>> (machines) [1] and the effort done to gather the information shared in
> >>>>> the replies [2], and the various acceptance tests added, is it
> >>>>> feasible to require for the next release that each new device/machine
> >>>>> is provided a test covering it?
> >>>>>
> >>>>> If no, what is missing?
> >>>>
> >>>> If a qtest is feasible, yes, I think we should require one for new
> >>>> devices. But what about machines - you normally need a test image for
> >>>> this. In that case, there is still the question where testing images
> >>>> could be hosted. Not every developer has a web space where they could
> >>>> put their test images onto. And what about images that contain non-free
> >>>> code?
> >>>
> >>> Yep, it isn't feasible to make this a hard rule.
> >>>
> >>> IMHO this is where a support tier classification comes into play
> >>>
> >>>  - Tier 1: actively maintained, qtest coverage available. Expected
> >>>            to work reliably at all times since every commit is CI
> >>>      tested
> >>>
> >>>   - Tier 2: actively maintained, no qtest coverage. Should usually
> >>>            work but regression may creep in due to reliance on the
> >>>      maintainer to manually test on adhoc basis
> >>>
> >>>   - Tier 3: not actively maintained, unknown state but liable to
> >>>             be broken indefinitely
> >>>
> >>> Tier 1 is obviously the most desirable state we would like everthing to
> >>> be at. Contributors will have to fix problems their patches cause as
> >>> they will be blocked by CI.
> >>>
> >>> Tier 2 is an admission that reality gets in the way. Ideally stuff in
> >>> this tier will graduate to Tier 1 at some point. Even if it doesn't
> >>> though, it is still valid to keep it in QEMU long term. Contributors
> >>> shouldn't gratuitously break stuff in these board, but if they do,
> >>> then the maintainer is ultimately responsible for fixing it, as the
> >>> contributors don't have a test rig for it.
> >>>
> >>> Tier 3 is abandonware. If a maintainer doesn't appear, users should
> >>> not expect it to continue to exist long term. Contributors are free
> >>> to send patches which break this, and are under no obligation to
> >>> fix problems in these boards. We may deprecate & delete it after a
> >>> while
> >>>
> >>>
> >>> Over time we'll likely add more criteria to stuff in Tier 1. This
> >>> could lead to some things dropping from Tier 1 to Tier 2. This is
> >>> OK, as it doesn't make those things worse than they already were.
> >>> We're just saying that Tier 2 isn't as thoroughly tested as we
> >>> would like it to be in an ideal world.
> >>
> >> I really like the idea of device support tiers codified directly in the
> >> QEMU codebase, to give upstream users some idea of which devices we
> >> expect to work and which we ... don't, really.
> >>
> >> Not every last device we offer is enterprise production ready, but we
> >> don't necessarily do a good job of explaining which devices fall into
> >> which categories, and we've got quite a few of them.
> >>
> >> I wonder if a 2.5th tier would be useful; something like a "hobbyist"
> >> tier for pet project SoC boards and the like -- they're not abandoned,
> >> but we also don't expect them to work, exactly.
> >>
> >> Mild semantic difference from Tier 3.
> > 
> > I guess I was thinking such hobbyist stuff would fall into tier 2  if the
> > hobbyist maintainer actually responds to fixing stuff, or tier 3 if they
> > largely aren't active on the mailing list responding to issues/questions.
> > 
> > We add have a 4 tier system overall and put hobbyist stuff at tier 3,
> > and abandonware at tier 4.
> > 
> > Probably shouldn't go beyond 4 tiers though, as the more criteria we add
> > the harder it is to clearly decide which tier something should go into.
> > 
> > The tier 1 vs 2 divison is clearly split based on CI which is a simple
> > classification to decide on.
> > 
> > The tier 2 vs 3 division is moderately clearly split based on whether
> > there is a frequently active maintainer.
> > 
> > We can probably squeeze in the 4th tier without too much ambiguity in
> > the classisfication if we think it is adding something worthwhile either
> > from our POV as maintainers, or for users consuming it.
> 
> Yes, I didn't mean to start watering it down into a 1,380 tier system
> that we're never able to properly utilize.
> 
> I was thinking more along the lines of:
> 
> - Device works and is well loved
> - Device works and is well loved (but we have to test manually)
> - Device doesn't work, but is well loved
>   (Use at your own peril, please file a bug report)
> - Device doesn't work, and is unloved
> 
> Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where
> things can shift from 1A to 1B as their test coverage allows, but it's
> not meant to indicate general status otherwise.

Yeah 1A/1B would be fairly effective.

> Mostly, I would just like some way for users to avoid accidentally
> running tier 2 or 3 devices /by accident/, or the ability to compile
> QEMU versions that only allow tier 1 devices to be used.
> 
> It's all arbitrary -- but I think we agree more than not! I'd love to
> have a list of first-class boards and devices that we promise to test
> and have working.

Yes, I think we're basically in agreement on the both the goal and
way to achieve it.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|