Re: [PATCH v4] introduce vfio-user protocol specification

[Felipe Franciosi]

Hi,

> On Oct 13, 2020, at 10:30 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Fri, Oct 02, 2020 at 10:14:23AM +0000, Felipe Franciosi wrote:
>> 
>> 
>>> On Sep 30, 2020, at 3:24 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>> 
>>> On Tue, Sep 29, 2020 at 09:21:54AM -0700, John G Johnson wrote:
>>>>> On Sep 29, 2020, at 3:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>>> 
>>>>> On Mon, Sep 28, 2020 at 09:58:37AM +0000, Thanos Makatos wrote:
>>>>>>> It should be accompanied by a test in tests/. PCI-level testing APIS for
>>>>>>> BARs, configuration space, interrupts, etc are available in
>>>>>>> tests/qtest/libqos/pci.h. The test case needs to include a vfio-user
>>>>>>> device backend interact with QEMU's vfio-user-pci implementation.
>>>>>> 
>>>>>> We plan to use a libmuser-based backend for testing. This, I suppose, 
>>>>>> will
>>>>>> make libmuser a dependency of QEMU (either as a submodule or as a 
>>>>>> library),
>>>>>> which for now can be disabled in the default configuration. Is this 
>>>>>> acceptable?
>>>>> 
>>>>> If there are no other dependencies and libmuser supports all host
>>>>> operating systems that QEMU's -device vfio-user supports, then I think
>>>>> it's a good idea to use libmuser for at least one in-tree test in QEMU.
>>>>> 
>>>>>>> Also please let us know who is working on what so additional people can
>>>>>>> get involved in areas that need work!
>>>>>> 
>>>>>> Swapnil and I will be working on libmuser and the test in QEMU, John and
>>>>>> the mp-qemu folks will be working on the patches for implementing
>>>>>> --device vfio-user-pci.
>>>>> 
>>>>> Great!
>>>>> 
>>>>> John: Will mpqemu use libmuser to implement the remote PCI host
>>>>> controller?
>>>>> 
>>>> 
>>>> 
>>>>    The vfio-user-pci plan is to use libmuser on the server side.
>>> 
>>> Okay. Using libmuser in tests seems like a good choice in that case.
>>> 
>>> We'll need to figure out the details of how to do it because the
>>> traditional shared library dependency approach is not well-suited to
>>> in-development code. It would involve shipping libmuser distro packages
>>> so QEMU's build system can declare a library dependency (with details
>>> provided in a pkg-config file).
>>> 
>>> Here are approaches that are better for in-development libraries:
>>> 1. Keep the libmuser code in qemu.git.
>>> 2. A copy of libmuser in qemu.git with changes being sent upstream
>>>  (allows more flexibility in case QEMU-specific issues require
>>>  experimentation).
>>> 3. Git submodules.
>>> 
>>> #1 if you're happy to use the QEMU development process for merging
>>> libmuser code then it's easiest to officially host the code in qemu.git.
>>> libmuser gets a subdirectory in the qemu.git tree and you (the
>>> maintainers) send pull requests. A libmuser library build target
>>> provides installable static and shared libraries so external
>>> applications can link against libmuser too. The big advantage here is
>>> that QEMU can instantly use the latest libmuser code changes.
>> 
>> I think there's a couple of limitations here which we should keep in mind.
>> 
>> 1. Does putting it in qemu.git precludes it being BSD-3?
>> There's been evidence of people using (or at least trying out) muser
>> from where it currently lives. That doesn't mean we can't move it, but
>> I'm wondering if it means we have to make it GPL.
> 
> The 3-clause BSD license is compatible with the GPL according to
> Wikipedia:
> https://en.wikipedia.org/wiki/BSD_licenses

Ah, ok. That's not an issue then.

> 
>> 2. What about other projects that need libmuser code?
>> What worries me more is projects like SPDK/DPDK wanting to link
>> against the library and having to clone the entire QEMU repo as a
>> submodule. That sounds a lot more expensive than option 3 and probably
>> have further complications if they aren't GPL.
> 
> In the early stages where the vfio-user protocol and library interfaces
> might need changes it will be hard to use it from multiple applications
> without compatibility issues. If SPDK/DPDK are communicating with QEMU
> using a cutting-edge library then they probably need to build QEMU from
> source anyway. ISTR they also maintain a QEMU fork? So maybe it's not a
> big issue for them.

A few things on this:

Per Daniel's response on the other fork of the thread, I think we can
develop faster if we're not depending on qemu-devel. With the right
set of maintainers on the project, there's arguably more flexibility
in working at a restricted set of code. QEMU can then update the
submodule when a "checkpoint" is ready.

Also, I don't see why SPDK/DPDK would need to build QEMU from source
for their implementations. That is, they will need an up-to-date QEMU
for enabling their development, but they don't provide a QEMU to their
users; they just have to wait for a release.  I talked to Ben Walker
(SPDK), and he confirmed SPDK doesn't really maintain a fork of QEMU.
Apparently there was one when they were doing vhost-user-nvme, but
that's now abandoned.

If they had to add the entire QEMU as a submodule, it could
potentially be an issue, right? That's a lot of code just to get some
headers.

> 
>>> 
>>> #2 works best if the library is a small (just a few source files) with
>>> no fancy build system requirements.
>> 
>> The risk here is that they go out of sync. There's the same (or even
>> more) maintenance burden as point 3 below, with the added risk that
>> someone could patch the files and make cherry-picks non-trivial.
>> 
>>> 
>>> #3 is used in QEMU for several other components. Submodules are a pain
>>> to sync (requires sending a qemu.git patch to move to a new commit ID),
>>> so this isn't good for a dependency that moves quickly.
>> 
>> I argue this is no worse than option 2. It's what I think aligns best,
>> but let's keep weighing pros/cons and come to a conclusion together.
>> The list of maintainers for muser.git should be extended to include
>> more QEMU stakeholders and probably other projects that will use it
>> (as) heavily. The topic has been raised in SPDK's Slack team on
>> whether the client library should live in a repo of its own (eg.
>> libvfio-user.git). Given the reference implementation is in libmuser,
>> I still think muser.git is accurate (but can easily be persuaded
>> otherwise).
> 
> Me too, no solution is perfect. My thoughts about developing it within
> qemu.git for now is that this will make protocol and library interface
> changes easy. It will also encourage applications (DPDK/SPDK) to build
> against a matching QEMU so that there are no compatibility problems at
> the protocol or library level while the code is still heavily under
> development.

I suppose that challenge is up to the vfio-user/libmuser maintainers
to resolve by tagging rc versions or something along those lines.
Those would be the ones we can add to QEMU master as the work
progresses.

Thoughts?
F.

> 
> Stefan