Re: when to use virtio (was Re: [PATCH v4 0/8] Introduce the microvm machine type)

[David Hildenbrand]

On 25.09.19 12:19, Paolo Bonzini wrote:
> This is a tangent, but I was a bit too harsh in my previous message (at
> least it made you laugh rather than angry!) so I think I owe you an
> explanation.

It's hard to make me really angry, you have to try better :) However,
after years of working on VMs, VM memory management and Linux MM, I
learned that things are horribly complicated - it's not obvious so I
can't expect all people to know what I learned.

> 
> On 25/09/19 10:44, David Hildenbrand wrote:
>> I consider virtio the silver bullet whenever we want a mature
>> paravirtualized interface across architectures. And you can tell that
>> I'm not the only one by the huge amount of virtio device people are
>> crafting right now.
> 
> Given there are hardware implementation of virtio, I would refine that:
> virtio is a silver bullet whenever we want a mature ring buffer
> interface across architectures.  Being friendly to virtualization is by
> now only a detail of virtio.  It is also not exclusive to virtio, for
> example NVMe 1.3 has incorporated some ideas from Xen and virtio and is
> also virtualization-friendly.
> 
> In turn, the ring buffer interface is great if you want to have mostly
> asynchronous operation---if not, the ring buffer is just adding
> complexity.  Sure, we have the luxury of abstractions and powerful
> computers that hide most of the complexity, but some of it still lurks
> in the form of race conditions.
> 
> So the question for virtio-mem is what makes asynchronous operation
> important for memory hotplug?  If I understand the virtio-mem driver,
> all interaction with the virtio device happens through a work item,
> meaning that it is strictly synchronous.  At this point, you do not need
> a ring buffer, you only need:

So, the main building pieces virtio-mem uses as of now in the virtio
infrastructure are the config space and one virtqueue.

a) A way for the host to send requests to the guest. E.g., request a
certain amount of memory to be plugged/unplugged by the guest. Done via
config space updates (e.g., similar to virtio-balloon
inflation/deflation requests).
b) A way for the guest to communicate with the host. E.g., send
plug/unplug requests to plug/unplug separate memory blocks. Done via a
virtqueue. Similar to inflation/deflation of pages in virtio-balloon.

Requests by the host via the config space are processed asynchronously
by the guest (again, similar to - say - virtio-balloon). Guest requests
are currently processed synchronously by the host.

Guest: Can I plug this block?
Host: Sorry, No can do.

Can't tell if there might be extensions (if virtio-mem ever comes to
life ;) ) that might make use of asynchronous communication. Especially,
there might be asynchronous/multiple guest->host requests at some point
(e.g., "I'm nearly out of memory, please send help").

So yes, currently we could live without the ring buffer. But the config
space and the virtqueue are real life-savers for me right now :)

> 
> - a command register where you write the address of a command buffer.
> The device will do DMA from the command block, do whatever it has to do,
> DMA back the results, and trigger an interrupt.
> 
> - an interrupt mechanism.  It could be MSI, or it could be an interrupt
> pending/interrupt acknowledge register if all the hardware offers is
> level-triggered interrupts.
> 
> I do agree that virtio-mem's command buffer/DMA architecture is better
> than the more traditional "bunch of hardware registers" architecture
> that QEMU uses for its ACPI-based CPU and memory hotplug controllers.
> But that's because command buffer/DMA is what actually defines a good
> paravirtualized interface; virtio is a superset of that that may not be
> always a good solution.
> 

I completely agree to what you say here, virtio comes with complexity,
but also with features (e.g., config space, support for multiple queues,
abstraction of transports).

Say, I would only want to expose a DIMM to the guest just like via ACPI,
virtio would clearly not be the right choice.

-- 

Thanks,

David / dhildenb