Re: Building QEMU as a shared library

[Daniel P . Berrangé]

On Wed, Dec 15, 2021 at 09:45:56AM +0000, Stefan Hajnoczi wrote:
> On Wed, Dec 15, 2021 at 08:18:53AM +0000, Amir Gonnen wrote:
> > Before sending a patch, I would like to check if it's of interest to the 
> > community.
> > 
> > My goal is to simulate a mixed architecture system.
> > Today QEMU strongly assumes that the simulated system is a *single 
> > architecture*.
> > Changing this assumption and supporting mixed architecture in QEMU proved 
> > to be
> > non-trivial and may require significant development effort. Common code 
> > such as
> > TCG and others explicitly include architecture specific header files, for 
> > example.
> 
> Hi Amir,
> Simulating heterogenous machines comes up from periodically. So far no
> one has upstreamed a solution but there is definitely interest.
> 
> I suggest going ahead and posting the code even if it's not cleaned up.
> 
> > A possible solution, discussed on 
> > https://stackoverflow.com/q/63229262/619493 is to
> > separate the simulation to multiple processes (as done by Xilinx) and to 
> > use some form
> > of Interprocess Communication channel between them.
> > Such solution has several disadvantages:
> > 
> > - Harder to synchronize simulation between processes
> > - Performance impact of Interprocess Communication
> > - Harder to debug, profile and maintain
> > 
> > Instead, I would like to suggest a new approach we use at Neuroblade to 
> > achieve this:
> > Build QEMU as a shared library that can be loaded and used directly in a 
> > larger simulation.
> > Today we build qemu-system-nios2 shared library and load it from 
> > qemu-system-x86_64 in order
> > to simulate an x86_64 system that also consists of multiple nios2 cores.
> > In our simulation, two independent "main" functions are running on 
> > different threads, and
> > simulation synchronization is reduced to synchronizing threads.
> > 
> > To achieve this, I needed to do the following changes in QEMU:
> > 
> > 1. Avoid Glib global context. Use a different context (g_main_context_new) 
> > for each QEMU instance.
> > 2. Change meson.build to build QEMU as a shared library (with PIC enabled 
> > for static libraries)
> > 3. Define a C API for the library and export it (with a 
> > -Wl,--version-script)

I'm curious what kind of scope the C API has and whether it is likely to
conflict with any general plans we have for future evolution to QEMU's
design, especially around configuration / CLI.

In your example where the client loading the QEMU shared library is
QEMU itself, the licensing doesn't become an issue, as everything
is within the same project codebase.

If the shared library were exposed to consumers outside of the
QEMU project, then licensing is a significant stumbling block,
as any consumer has to be GPL-v2-only compatible. It is pretty
easy to get into a license incompatibility situation with this
requirement. I fear alot of people would just ignore this as an
"inconvenience" and knowingly use incompatible code.

Both of these issues could be avoided if we make any shuared
library a QEMU-internal thing, by not installing any headers
and doing something to ensure its ABI changes on every build.

> > These changes seem enough for simulating mixed architecture system on a 
> > single process.
> > 
> > If this approach sounds useful, I'll be happy to send patches.
> > I'd appreciate if you could provide your feedback!
> 
> I'm curious how much synchronization and IPC there is between the QEMU
> shared libraries? I would have guessed that the pain of making
> communication work efficiently between processes would be less than the
> pain of solving global state bugs related to shared libraries within a
> single process.

We've already got ability to have device backends run in separate
processes with vhost-user, and have various other efforts underway
to support more generalized out of tree device implementations.
With this direction in mind, it feels conceptually desirable  if
we are able to support heterogenous CPU emulation using co-operating
processes too, as opposed to a monolithic process architecture.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|