Am 04.05.2022 um 21:38 hat John Snow geschrieben:
> Howdy!
>
> So, I want to finally delete python/qemu/qmp from qemu.git, and this
> creates a small problem -- namely, iotests needs access to it in order
> to run the python-based tests.
>
> What I think needs to happen is that we create a virtual environment
> that installs python/qemu/. The reason this cannot be done with
> PYTHONPATH alone anymore is because the qmp package itself won't be
> there anymore, we need an installer like `pip` to actually fetch it
> for us and put it somewhere. (i.e., we need to process the
> dependencies of python/qemu now and can't treat it as a pre-installed
> location.)
>
> Avocado tests are already creating a venv for the purposes of
> installing and running Avocado. We can amend e.g. "../../python" to
> tests/requirements.txt and the Avocado environment is A-OK good-to-go.
> The Makefile magic for avocado tests creates a venv-per-build.
"per build" sounded pretty bad, because I thought it meant building a
new venv every time I run either 'make' or the tests. This would
obviously be very noticable for short-running tests like some iotests.
But fortunately this is not what it does, it keeps the venv around in
the build directory. So it's only per build directory really, which I
guess is fine.
Whoops, yeah. I meant per build directory. In contrast to the Python unit tests themselves, which create some test venvs tied directly to the source directory and are build-agnostic.
> It seems to work well enough. One thing to note here is that the
> supported invocation for avocado tests is only through the Makefile,
> which handles creating and entering the venv to make the command
> seamless.
>
> iotests, however, manages its own execution environment with
> testenv.py, and we support running iotests from outside of the
> Makefile, for example by going to $build/tests/qemu-iotests and
> running ./check.
Yes, and I agree that this is important.
Figured as much. Not plotting to take this away, I promise. Just getting my requirements straight before I spend time concocting something.
> Now ... I could update testenv.py to be smart enough to create and
> enter a python venv, and have even prototyped this. It seems to work
> pretty well! This approach seemed like the least invasive to how
> iotests are expected to be run and used. But a downside with this
> approach is that now avocado tests and iotests are each managing their
> own python venv. Worse, vm-tests and device-crash-test are still
> unhandled entirely.
Is there a reason why testenv.py couldn't enter just the shared venv in
build/tests/venv?
It can, but it'd have to be made first so it can enter it. I figured this would only happen "on-demand", like how check-avocado works, so I'd need some way for iotests and the Makefile to share the venv creation code, which is certainly an option.
If not, I guess it would be enough if iotests just checks that the venv
exists and all of the dependencies are there in the right version and
error out if not, telling the user to run 'make check-venv'.
I will spend unreasonable engineering hours avoiding this, if only for pride. I want everything to be as seamless and easy as it's always been.
Or actually, it could just unconditionally run 'make check-venv' by
itself, which is probably easier to implement than checking the
dependencies and more convenient for the user, too.
Oh, that's one way to get them to share the venv-creation pathway. Hadn't occurred to me, but this might be easy to do.
(One more detail: 'make check-venv GIT_SUBMODULES_ACTION=ignore' seems
to make it pretty much instantaneous if the venv is current, but leaving
the submodule mechanism enabled adds a noticable delay.)
Noted.
> I'd like to find a solution where I create a unified python testing
> venv tied to the build shared by avocado, iotests, vm-tests and
> device-crash-test. I'm not completely sure how exactly I'll manage
> that right now, but I wanted to throw this out there in case there are
> some requirements I might be overlooking.
>
> I think vm-tests and avocado-tests can both have a venv created for
> them and activated before the test runs. device-crash-test I believe
> will need a script change in the gitlab ci yaml. iotests is somewhat
> unique in that it needs to run both by manual invocation and from
> makefile invocations. If I want a shared VM between all of these, I'll
> need to isolate the create-and-enter-venv logic somewhere where it can
> be shared both inside and outside of a Makefile.
If just calling 'make' isn't an option, then moving that part out into a
separate script probably wouldn't be too hard either. But 'make' has the
advantage that it already contains the check if the venv is already
there and requirements.txt hasn't changed, which you would have to
replicate otherwise.
Yeah, exactly.
Thanks for the ideas!
As of now, I've got a branch where I've moved vm-tests over onto the same venv with avocado-tests. If I can bridge the gap to iotests, I'll be pretty happy.