Re: docker: how to use it when developing on QEMU?

[John Snow]

On 9/24/19 11:25 AM, Philippe Mathieu-Daudé wrote:
> Recently more developers are enthusiast to use Docker/Podman,
> and have been confused by the different configurations currently in the
> QEMU repository.
> 

I think it's good evidence we need to improve the abstractions for this
testing module. It's not easy to know what to do without getting direct
feedback from the maintainers directly, which won't scale.

> There are at least 3 kind of categories I use:
> 
> 1/ Image used to build QEMU
> 
> These images should be restricted/updated to our "supported targets".
> They are useful to (cross)build QEMU on variety of
> host/target/distributions/distrib_versions.
> 
> Example:
> 
> - cross build Cris binary using the Fedora 30 toolchain on a Ubuntu
> 18.04 x86_64.
>   host:Ubuntu18.04/x86_64 docker_image:fedora-cris-cross
> 

Is this supposed to be a command invocation? Can we see full command
lines instead?

> - cross build MinGW64 binary using Debian 9 MXE toolchain on a Ubuntu
> aarch64:
>   host:Ubuntu18.04/x86_64 docker_image:debian-win64-cross
> 
> An image can not be meant to use on a daily basis, but to avoid
> regression previous to release (I'd run them only on release candidate).
> 

Do you mean to say "These images are not meant to be used on a daily
basis, but instead on occasion to prevent regressions during release
candidate testing." ?

They can build QEMU, but I assume they can't run any tests. Is there a
special "build-only" target that you can invoke from the Makefile to get
these to run?

> Example: building QEMU for the Gentoo PlayStation2 port:
> https://www.mail-archive.com/address@hidden/msg574468.html
> 
> 2/ Image used to build test program used by QEMU
> 

I think this is the category most people are wanting to get their hands
on, usually.

> These images provide enough to build binaries you can then use to test
> QEMU. If you want to build more of these binaries, there is probably a
> better way. Here we are only interested in testing.
> 
> Example:
> 
> - Test PowerPC Linux-user binaries with qemu-powerpc-linux-user
>   docker_image:debian-powerpc-user-cross
> 
> - Build EDK2 payload for Virt/AArch64
>   It currently doesn't build with Fedora 30 and I'v to use a Fedora 29
> image.
> 
> Another case I had is when I tried to build a kernel for the Mipssim
> machine (supported by QEMU). The Linux kernel code has been removed, so
> I had to checkout an old kernel which is not buildable with my recent
> host GCC. Using a docker based on a very old distribution worked. Anyway
> Thomas Huth found it is easier to use buildroot for pre-3.6 kernels.
> 
> Similarly, I am testing QEMU port from Stefan Weil, and he shared a
> working binary supporting the MIPS AR7 target. To be able to use this
> QEMU I use Debian Lenny and set
> DEB_URL=https://snapshot.debian.org/archive/debian/20091004T111800Z.
> Yes, this will instanciate a Debian from 10 years ago.
> 
> 3/ Bisecting
> 
> Another of my docker uses is when bisecting before QEMU v3. I use image
> using snapshot slighly older than the QEMU release, so my bisect script
> can run without worrying about the library API incompatibilities or
> newer GCC warnings.
> 

Which images, tests, or invocations are useful for this?

> So not all image have the same use. While they might not be useful to
> build the latest QEMU, there are still useful for day-to-day development.
> 

I would say that this has grown beyond the abstractions presented by the
Docker makefile, which makes it hard to figure out how to use any of the
tools in this directory.

> Anyhow I agree we should document that better. Maybe the wiki is a good
> starting point.
> 

That might be a good place to organize your thoughts, but the source of
truth must be the Makefile help invocation.

> Regards,
> 
> Phil.
> 


We should split the images into a few categories in the makefile
directly, in a programmatic way;

1. Partial Images

These images are not meant for building or testing anything directly.
They are dependencies; they should be better distinguished in the
Makefile help.

2. Unrunnable Images

These images would otherwise not be considered partial, but are
conditionally unrunnable depending on the architecture of the host system.

3a. Images that can build QEMU

Runnable images; but only useful for building QEMU and/or its tools. A
lot of the cross-compiler images might fit into this category.

3b. Images that can test QEMU

Runnable images; but can run iotests, make check, and so on. We expect
all of the tests defined, in general, to work for this image.


Does that sound about right?

--js