Re: [PATCH v2 03/15] python: add VERSION file

[John Snow]

On 10/19/20 6:02 AM, Daniel P. Berrangé wrote:
> On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote:
> > On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote:
> > > Python infrastructure as it exists today is not capable reliably of
> > > single-sourcing a package version from a parent directory. The authors
> > > of pip are working to correct this, but as of today this is not possible
> > > to my knowledge.
> > >
> > > The problem is that when using pip to build and install a python
> > > package, it copies files over to a temporary directory and performs its
> > > build there. This loses access to any information in the parent
> > > directory, including git itself.
> > >
> > > Further, Python versions have a standard (PEP 440) that may or may not
> > > follow QEMU's versioning. In general, it does; but naturally QEMU does
> > > not follow PEP 440. To avoid any automatically-generated conflict, a
> > > manual version file is preferred.
> > >
> > >
> > > I am proposing:
> > >
> > > - Python core tooling synchronizes with the QEMU version directly
> > >    (5.2.0, 5.1.1, 5.3.0, etc.)
> > >
> > > - In the event that a Python package needs to be updated independently
> > >    of the QEMU version, a pre-release alpha version should be preferred,
> > >    but *only* after inclusion to the qemu development or stable branches.
> > >
> > >    e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's
> > >    release.
> > >
> > > - The Python core tooling makes absolutely no version compatibility
> > >    checks or constraints. It *may* work with releases of QEMU from the
> > >    past or future, but it is not required to.
> > >
> > >    i.e., "qemu.core" will always remain in lock-step with QEMU.
> > >
> > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp
> > >    and begin indepedently versioning such a package separately from the
> > >    QEMU version it accompanies.
> >
> > I think this need to be considered very carefully.
> >
> > I'm not overly familiar with the Python ecosystem but it would appear
> > that, despite PEP 440 not mandating this, many (most?) of the
> > packages uploaded to PyPi are using semantic versioning.
>
>    
> https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme
>
> Semver is the recommended approach, but they explicitly list date
> based versioning as a valid alternative
>
>    "Semantic versioning is not a suitable choice for all projects,
>     such as those with a regular time based release cadence and a
>     deprecation process that provides warnings for a number of
>     releases prior to removal of a feature."
>
> That paragraph describes QEMU's scenario.
>
> NB, historically we've made arbitrary changes to the python code
> since it was not considered public API. If we make it official
> public API, then we would actually need to start following our
> deprecation process for the python code too.

I think our deprecation process is not tightly compatible with how 
Python programmers at-large expect packages to work. Semver is more or 
less the norm, despite the fact that it isn't explicitly required.

setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc 
often hinge on a major version, e.g.

qemu >= 5.0
qemu >= 3.0, < 6.0

would both be common forms of describing a requirement.

Pinning specific versions is considered bad form, but in the context of 
releasing a package, I often see maintainers hedging their bets and 
preventing upgrades across a major version line.

For that reason I am a little weary of adopting the deprecation policy 
as it exists in QEMU directly, and would propose a modification for my 
purposes here:

- Features must be marked as deprecated
- They must remain in a deprecated state for [at least] 2 releases
- Deprecated features may not be removed until a major version change.

In practice, this modification is a change from "2 releases" to "at 
least 2".

However, I didn't intend to pay any mind to the deprecation policy 
"yet", as I have the package metadata listing the package status as 
"Alpha", see below:

> > With that in mind, I think it would be unwise for qemu.* not to do
> > the same; in particular, using a version number that's not <1.0.0 for
> > a package that is very much in flux will almost certainly break
> > people's expectations, and is also not something that you can easily
> > take back at a later time.
>
> I don't think it is that big a deal, and there is clear benefit to
> having the python code version match the QEMU version that it is
> the companioon to.

Do you think it's fine if I start versioning at, say, "0.5.2", I could 
ignore a deprecation policy for now? I have a lot of changes I want to 
make and expect a lot of breaking changes very quickly.

I just wanted to try -- somehow -- to conjure up a relationship to the 
QEMU package it's designed to work with.

> Ultimately the versioning scheme just impacts on the version string
> conditionals people list for their dependancies. Apps consuming QEMU
> can handle any of the version schemes without much difference.
>
> Regards,
> Daniel

Thanks for your input. This is the trickiest part of the process for me.

I believe there is value in distributing these tools for other 
developers to help them prototype and experiment with new features, but 
realize it's a tightrope walk because we're flying dangerously close to 
providing a management utility that needs to care about cross-version 
compatibility and so on.

I have no interest in providing stringent cross-version compatibility, 
but at the same time, libraries like QMP are not really at risk of 
changing all that much, actually. It will *probably* work for most QEMU 
versions.

I have some further patches where I clean up the scripts in 
./scripts/qmp and move them to ./python/qemu/qmp -- and that work is 
making me consider a rework to this series. I think that rework matches 
the spirit of an earlier suggestion of yours:

- move ./python/qemu/core/qmp.py to ./python/qemu/qmp/qmp.py
- (in a follow-up series) move ./scripts/qmp/* to ./python/qemu/qmp/*
- rename ./python/qemu/core/ to ./python/qemu/machine
    (move accel.py, machine.py and qtest.py to a 'machine' pkg.)


In effect, we'd then have:

qemu.machine -- primarily a test interface for QEMU instances
qemu.qmp -- fairly ageless QMP tools/lib for talking to QEMU

and the different levels of support and "use at your own risk" become 
slightly more clear between the two packages.

--js