Re: RFC: use VFIO over a UNIX domain socket to implement device offloading

[Felipe Franciosi]

Hi,

> On Apr 30, 2020, at 4:20 PM, Thanos Makatos <address@hidden> wrote:
> 
>>>> More importantly, considering:
>>>> a) Marc-André's comments about data alignment etc., and
>>>> b) the possibility to run the server on another guest or host,
>>>> we won't be able to use native VFIO types. If we do want to support that
>>>> then
>>>> we'll have to redefine all data formats, similar to
>>>> https://urldefense.proofpoint.com/v2/url?u=https-
>>>> 3A__github.com_qemu_qemu_blob_master_docs_interop_vhost-
>>>> 
>> 2Duser.rst&d=DwIFAw&c=s883GpUCOChKOHiocYtGcg&r=XTpYsh5Ps2zJvtw6
>>>> 
>> ogtti46atk736SI4vgsJiUKIyDE&m=lJC7YeMMsAaVsr99tmTYncQdjEfOXiJQkRkJ
>>>> W7NMgRg&s=1d_kB7VWQ-
>> 8d4t6Ikga5KSVwws4vwiVMvTyWVaS6PRU&e= .
>>>> 
>>>> So the protocol will be more like an enhanced version of the Vhost-user
>>>> protocol
>>>> than VFIO. I'm fine with either direction (VFIO vs. enhanced Vhost-user),
>>>> so we need to decide before proceeding as the request format is
>>>> substantially
>>>> different.
>>> 
>>> Regarding the ability to use the protocol on non-AF_UNIX sockets, we can
>>> support this future use case without unnecessarily complicating the
>> protocol by
>>> defining the C structs and stating that data alignment and endianness for
>> the
>>> non AF_UNIX case must be the one used by GCC on a x86_64 bit machine,
>> or can
>>> be overridden as required.
>> 
>> Defining it to be x86_64 semantics is effectively saying "we're not going
>> to do anything and it is up to other arch maintainers to fix the inevitable
>> portability problems that arise".
> 
> Pretty much.
> 
>> Since this is a new protocol should we take the opportunity to model it
>> explicitly in some common standard RPC protocol language. This would have
>> the benefit of allowing implementors to use off the shelf APIs for their
>> wire protocol marshalling, and eliminate questions about endianness and
>> alignment across architectures.
> 
> The problem is that we haven't defined the scope very well. My initial 
> impression 
> was that we should use the existing VFIO structs and constants, however 
> that's 
> impossible if we're to support non AF_UNIX. We need consensus on this, we're 
> open to ideas how to do this.

Thanos has a point.

From https://wiki.qemu.org/Features/MultiProcessQEMU, which I believe
was written by Stefan, I read:

> Inventing a new device emulation protocol from scratch has many
> disadvantages. VFIO could be used as the protocol to avoid reinventing
> the wheel ...

At the same time, this appears to be incompatible with the (new?)
requirement of supporting device emulation which may run in non-VFIO
compliant OSs or even across OSs (ie. via TCP or similar).

We are happy to support what the community agrees on, but it seems
like there isn't an agreement. Is it worth all of us jumping into
another call to realign?

Cheers,
F.