Re: [RFC 0/8] virtio: Improve boot time of virtio-scsi-pci and virtio-blk-pci

[Stefan Hajnoczi]

On Thu, Mar 25, 2021 at 04:07:27PM +0100, Greg Kurz wrote:
> Now that virtio-scsi-pci and virtio-blk-pci map 1 virtqueue per vCPU,
> a serious slow down may be observed on setups with a big enough number
> of vCPUs.
> 
> Exemple with a pseries guest on a bi-POWER9 socket system (128 HW threads):
> 
> 1             0m20.922s       0m21.346s
> 2             0m21.230s       0m20.350s
> 4             0m21.761s       0m20.997s
> 8             0m22.770s       0m20.051s
> 16            0m22.038s       0m19.994s
> 32            0m22.928s       0m20.803s
> 64            0m26.583s       0m22.953s
> 128           0m41.273s       0m32.333s
> 256           2m4.727s        1m16.924s
> 384           6m5.563s        3m26.186s
> 
> Both perf and gprof indicate that QEMU is hogging CPUs when setting up
> the ioeventfds:
> 
>  67.88%  swapper         [kernel.kallsyms]  [k] power_pmu_enable
>   9.47%  qemu-kvm        [kernel.kallsyms]  [k] smp_call_function_single
>   8.64%  qemu-kvm        [kernel.kallsyms]  [k] power_pmu_enable
> =>2.79%  qemu-kvm        qemu-kvm           [.] memory_region_ioeventfd_before
> =>2.12%  qemu-kvm        qemu-kvm           [.] 
> address_space_update_ioeventfds
>   0.56%  kworker/8:0-mm  [kernel.kallsyms]  [k] smp_call_function_single
> 
> address_space_update_ioeventfds() is called when committing an MR
> transaction, i.e. for each ioeventfd with the current code base,
> and it internally loops on all ioventfds:
> 
> static void address_space_update_ioeventfds(AddressSpace *as)
> {
> [...]
>     FOR_EACH_FLAT_RANGE(fr, view) {
>         for (i = 0; i < fr->mr->ioeventfd_nb; ++i) {
> 
> This means that the setup of ioeventfds for these devices has
> quadratic time complexity.
> 
> This series introduce generic APIs to allow batch creation and deletion
> of ioeventfds, and converts virtio-blk and virtio-scsi to use them. This
> greatly improves the numbers:
> 
> 1             0m21.271s       0m22.076s
> 2             0m20.912s       0m19.716s
> 4             0m20.508s       0m19.310s
> 8             0m21.374s       0m20.273s
> 16            0m21.559s       0m21.374s
> 32            0m22.532s       0m21.271s
> 64            0m26.550s       0m22.007s
> 128           0m29.115s       0m27.446s
> 256           0m44.752s       0m41.004s
> 384           1m2.884s        0m58.023s

Excellent numbers!

I wonder if the code can be simplified since
memory_region_transaction_begin/end() supports nesting. Why not call
them directly from the device model instead of introducing callbacks in
core virtio and virtio-pci code?

Also, do you think there are other opportunities to have a long
transaction to batch up machine init, device hotplug, etc? It's not
clear to me when transactions must be ended. Clearly it's necessary to
end the transaction if we need to do something that depends on the
MemoryRegion, eventfd, etc being updated. But most of the time there is
no immediate need to end the transaction and more code could share the
same transaction before we go back to the event loop or vcpu thread.

Stefan

signature.asc
Description: PGP signature