Re: Integrating QOM into QAPI

[Christophe de Dinechin]

> On 26 Jan 2020, at 17:47, Paolo Bonzini <address@hidden> wrote:
> 
> On 26/01/20 10:11, Marc-André Lureau wrote:
>>> I’m still puzzled as to why anybody would switch to something like
>>> GObject when there is C++.
>> C++ is another level of complexity.
>> 
>> Replacing QOM with GObject would mainly bring us a more solid type
>> system with better tooling/features, gobject-introspection support,
>> and remove the burden of having our own OO from QEMU code base.
> 
> In fact, C++ doesn't solve any of the problems that either QOM or
> GObject try to solve.  (Neither does Rust for that matter).

It does not solve all of them _easily_. I do believe that any solution
to these problems can be engineered in C++, if only because C++
is essentially a superset of C. The question is whether the result can
be made easier / safer to use and generally more elegant. I believe
that the answer is yes, see proof at end of mail.

However, before going further, glib offers way more than gobject.
So there’s that too… And I’m not saying migrating to C++ is a good
idea, I’m just trying to evaluate the various options fairly.


> Nevertheless, there is no stupid question, only stupid answers, and I
> think Christophe's remark is an example of a common misconception.  In
> the hope of not making this a stupid answer, let my try to formulate
> succinctly what I think the differences are between QOM, GObject and the
> C++ object model:

Thank you for this remarkably non-stupid answer ;-) You have really
isolated, in very clear terms, the essence of the discussion.

> 
> - the C++ object model (at least "old-style" C++ with virtual functions
> and the like) provides you with _the intersection_ of what QOM and
> GObject try to solve.  This is what Marc-André calls "OO", and it's
> essentially virtual functions and dynamic casts.  It's a relatively
> small part of both QOM and GObject, and unfortunately a wheel that
> almost every large C program ends up reinventing.

This was the part I was pointing to in my initial comment.

C++ solves that basic “OO” stuff well, and goes a little beyond QOM
or GObject in offering IMO many other benefits, e.g. wrt type safety.

> 
> - Marc-André also described above what GObject provides: a fully
> introspectable type system and the tools so that _libraries_ can define
> _types that will be used from multiple programming languages_.

This kind of things has existed for a very long time. CORBA dates back
to 1991. Also, I’m not sure how important multiple programming languages
are for the qemu use-case. I believe that what matters more is remotely
accessible objects (e.g. over a socket), which in turn makes it almost
trivial to call from another language as long as you accept some kind
of serialization / deserialization process along the way.

GObject only seems marginally better for the “in same process” use case,
in the sense that it makes “objects” that can be used from any language,
indeed, but at the cost of being somewhat foreign and weird in all languages,
including C.

Look at the GClosure marshalling, for example, and compare it with the
example I give you at end of this email, and you tell me which one looks
easier and safer to use.

> 
> - QOM also provides a fully introspectable type system, but with a
> different focus: it's so that _objects_ can expose _properties that will
> be accessed from multiple channels_.

Exposing the properties and making them introspectable are the
fundamental feature we need to discuss. So agreement here.

What you have not explained to my satisfaction
yet is how GObject is a better starting point for re-creating a new
externally-accessible API than some kind of wrapper built in C++
and taking advantage of modern C++ features.


> Everything else in both GObject and QOM follows from this core purpose,
> and the differences between the two follow from the differences.  For
> example:
> 
> - GObject's focus on multiple programming languages:
> gobject-introspection, GClosure, support for non-object types (scalar
> and GBoxed)

How much of that is actually useful to create a new usable qemu API?

> 
> - QOM's focus on objects: dynamic properties, object tree, all types are
> classes

That, to me, looks fundamental, since not having it would require
a total re-architecture of the rest of qemu.

But it looks also somewhat trivial to implement in C++.
For example, obj[“id”] could return a PropertyAccessor<T>
that lets you read or write the object knowing that it is of type
T, so you could write:

        if (my_object[“id”] < 3) // Automatically checks the type

or 

        my_object[“id”] = 42;

The latter would call PropertyAccessor<int>::operator=(int), which
in turn would check if property “id” exists in my_object, if it has type
“int”, and so on.

Implementation-wise, a simple std::map of BaseProperty pointers,
where each would be an instance of some

        template<class T>
        class Property : public BaseProperty
        {
                operator T(); // get
                T& operator=(const T&); // set
        };

        template<class T, class Index>
        class PropertyAccessor
        {
                PropertyAccessor(PropertyOwner<T> &owner, Index &index) : 
owner(owner), index(index) {}
                operator T() { return owner[index]; }
                T& operator =(const T&val) { return owner[index] = val; }
        };

> - QOM's focus on properties: no introspection of methods

OK. So I’m giving a bad example below, because I’m introspecting
a method, but… ;-)

> - QOM's support for multiple channels: visitors

How different are visitors from iterators in C++? Sorry, showing my
ignorance here… 


I do not fully understand all the subtleties of QOM, nor all its use cases.
However, my own experience with “modern” C++ shows that it’s pretty
straightforward to get it to do extremely interesting introspective work.

As an illustration, which happens to be directly related to the problem
of interconnecting with another language, I would like to share the following
tidbit of code (https://github.com/c3d/xl/blob/master/src/runtime.cpp#L73).

Tree *  xl_evaluate(Scope *scope, Tree *tree)
// ----------------------------------------------------------------------------
//   Dispatch evaluation to the main entry point
// ----------------------------------------------------------------------------
{
    return MAIN->Evaluate(scope, tree);
}
XL_NATIVE(evaluate);

With this little XL_NATIVE macro, the compiler automatically makes the
function xl_evaluate available to the XL language as “evaluate”, with
the correct parameter sets. In particular:

- It inserts an object representing xl_evaluate in a list of all native 
functions
- It generates a function that generates the source-code interface,
  which in XL would look like `evaluate A:scope, B:tree as tree`,
  see record #28601 in the output below. Note that this includes 
  all the required type conversions (e.g. from `Tree *` to `tree`).
- It generates a function that generates a machine-level prototype
  using LLVM, i.e. it automatically all the LLVM calls required to declare
    Tree *xl_evaluate(Scope *scope, Tree *tree);
- It generates a function that acts as a wrapper around it and calls
  the prototype above, doing the necessary type conversions.

Here is an example of output at run-time:
> xl -tnative -O3
[…]
[26809 0.007432] native: Entering prototype for same_text shape 0x111a3f540 
[same_text A:tree, B:text as boolean]
[27021 0.007512] native: Shape 1 infix 0x111615d28 [tree] : 0x111ac8838 [A] = 
0x111a94af0 [A:tree]
[27213 0.007630] native: Return shape (one) 0x111615d28 [tree]
[27274 0.007651] native: Native shape 1 0x111a94aa8 [A:tree as tree] return 
0x111615d28 [tree]
[27447 0.007708] native: Entering prototype for stack_overflow shape 
0x111a3f510 [stack_overflow A:tree as tree]
[27595 0.007784] native: Shape 1 infix 0x111615c10 [scope] : 0x111ac87c8 [A] = 
0x111a94a60 [A:scope]
[27795 0.007844] native: Shape 2 infix 0x111615d28 [tree] : 0x111ac8790 [B] = 
0x111a94a18 [B:tree]
[27991 0.007897] native: Shape 2 (...) 0x111a94a60 [A:scope],0x111a94a18 
[B:tree] = 0x111a949d0 [A:scope, B:tree]
[28303 0.007985] native: Return shape (...) 0x111615d28 [tree]
[28364 0.008014] native: Native shape 2 0x111a94988 [A:scope, B:tree as tree] 
return 0x111615d28 [tree]
[28601 0.008077] native: Entering prototype for evaluate shape 0x111a3f4e0 
[evaluate A:scope, B:tree as tree]
[…]

What I believe makes my point is that this is achieved with 476 lines of C++
(admittedly not the simplest C++ on earth, but still not super-complex either)
If you are curious, this is done by 
https://github.com/c3d/xl/blob/master/include/native.h.

So in 500 lines of C++ code, we have something that automatically turns any C 
function into
a wrapper object that:
- Gives you introspected type information about the parameters and return type 
of the function
- Gives you directly-accessible methods that operate on that function in a 
type-safe way

The same approach could of course offer serialization, construction of objects, 
etc.

So to me, this proves that introspecting an API in C++ is not just “easy”, it 
is highly usable.


Cheers,
Christophe

PS: Again, I’m not necessarily advocating we start writing any C++ code.
I’m just trying to understand the value proposition of GObject, QOM, etc,
in order to make a more informed decision before doing anything.

> 
> Paolo
>