Re: Making QEMU easier for management tools and applications

[Kevin Wolf]

Am 07.01.2020 um 18:11 hat Christophe de Dinechin geschrieben:
> So I think that it might help, in the long run, to start defining the
> language in question in some abstract way, and then to have rules
> for how to transform that abstract language into concrete bindings.

I think this abstract language is QAPI. The problem is that we're not
even close to using QAPI for everything. Adding a new language on top of
QAPI instead isn't going to make the conversion process any faster.

> This definition itself is not obvious (at least not to me). For
> example, do we have, anywhere but in the C code, the specification
> of how one can add a disk to qemu, and what it means?
> Say, looking at qemu-options.def, how do I tell that -hda has
> anything to do with -device or -blockdev or -help?

BlockdevOptions in the QAPI schema is what tells you how it _really_
works. The connection to the various command line syntaxes isn't defined
in a declarative way because we don't have a QAPIfied command line yet.
I know that Markus wants to work on this, but I don't know how much time
he actually has to invest in it.

> I think that the following piece of code from vl.c is revealing:
> 
>             case QEMU_OPTION_hda:
>             case QEMU_OPTION_hdb:
>             case QEMU_OPTION_hdc:
>             case QEMU_OPTION_hdd:
>                 drive_add(IF_DEFAULT, popt->index - QEMU_OPTION_hda, optarg,
>                           HD_OPTS);
>                 break;
>             case QEMU_OPTION_blockdev:
>                 {
>                     Visitor *v;
>                     BlockdevOptionsQueueEntry *bdo;
> 
>                     v = qobject_input_visitor_new_str(optarg, "driver",
>                                                       &error_fatal);
> 
>                     bdo = g_new(BlockdevOptionsQueueEntry, 1);
>                     visit_type_BlockdevOptions(v, NULL, &bdo->bdo,
>                                                &error_fatal);
>                     visit_free(v);
>                     loc_save(&bdo->loc);
>                     QSIMPLEQ_INSERT_TAIL(&bdo_queue, bdo, entry);
>                     break;
>                 }
>             case QEMU_OPTION_drive:
>                 if (drive_def(optarg) == NULL) {
>                     exit(1);
>                 }
>                 break;
> 
> Here, we have three cases related to disks in a way or another,
> and three entirely different ways of doing things.

I would say two different ways because drive_add() is just a small
wrapper around drive_def() that overrides a few options.

Describing the semantics of the -drive way is hard. This is one of the
reasons why I would love to get rid of it and replace it with a new
user-friendly option that has a more direct mapping to the -blockdev
way, which in turn just is BlockdevOptions mapped 1:1 to the command
line.

> AFAICT, qemu already created several meta-languages to define
> several aspects of the API, from qemu-options.def to qapi-schema.json.
> But maybe at some point we need to go meta once more, and define
> a language defining the API from which we could automatically
> derive the various bindings, including FFI-style bindings for Rust and Go,
> as well as some internal data structures. Ideally, that meta-definition
> is something that could be shared between libvirt and qemu so that they
> literally speak the same language. Or that could be used to automatically
> build a REST interface.

I think adding an output for additional languages to the QAPI generator
shouldn't be too hard. It already creates multiple things from a single
schema (C data structures and command wrappers, schema introspection
data, documentation, and probably other things that I forgot).

libvirt already speaks QAPI, however without reusing the schema and the
generator from QEMU.

> A big issue, though, is that of compatibility. Doing the above starting
> from scratch does not seem that complicated. Doing it in a way that
> preserves a minimum of interoperability with earlier-generation
> software is another ordeal.

Indeed, this is the major reason why QAPI isn't as pervasive as it
should be.

> So I think that Daniel is right. We may need at some point to start
> a NEMU-style offshoot that does not attempt to be compatible,
> but explores describing an increasing surface of the API using a
> new meta-language from which we can generate, in a consistent
> way, at least:
> 
> - C bindings
> - Command-line options
> - Shell bindings (or “HMP”)
> - JSON schema or qom description
> - Bindings in other languages (Rust, Go, Python)
> - Networked versions of the API (socket, REST)
> - Client-side code e.g. for libvirt.
> - Serialization / deserialization, e.g. for configuration files
> - Documentation, including man page and API docs
> - Command-line help

I think the only thing in this list that can't obviously be covered
easily by QAPI is QOM. Or rather, it's covered by passing through
key=value lists without describing their structure - which, as far as I
understand, is mainly because QOM properties aren't necessarily static,
so we can't provide a statically defined interface for them. Probably
solvable in QEMU, but not without a major effort. In a fork that doesn't
care about compatibility, it should be easier.

> At the most fundamental level, I think we need to describe:
> 
> - Values, e.g. how we represent names, sizes, paths, etc, possibly
> with some user-friendly aspects, e.g. path shortcuts, memory units,
> spelling shortcuts (e.g. being able to consistently say -blo for -blockdev
> if that’s the shortest option that matches)

I don't think user-friendly shortcuts on the command line are "most
fundamental". Whether to accept -blo is an implementation detail of the
command line parser which translates a bunch of strings into QAPI
objects.

> - Relations, e.g. how we represent “contains”, “derives from”, “needs”,
> “one of”, “one or several”, “attaches to”…
> - States, e.g. how do we represent the machine configuration,
> or the desired new disk setting
> - Verbs, e.g. how we represent “add”, “connect”, “remove”, “find”,
> “start”, “notify”, etc. and how we describe the kind of input they need.
> - Possibly more subtle things like support for transactions, commit/rollback,
> i.e. “I want to add connect a virtual nic to some host vf, but if anything
> along the way fails, I’d like all the cleanup to happen automatically)

This sounds like a different approach from our current QAPI command set
(use a smaller set of operations that can work with a greater variety of
objects).

Does it actually provide more functionality, though?

Kevin