Re: [RFC PATCH 0/5] virtio-net: Introduce LM early load

[Eugenio Perez Martin]

On Mon, Sep 18, 2023 at 6:51 AM Yajun Wu <yajunw@nvidia.com> wrote:
>
> This series of patches aims to minimize the downtime during live migration of 
> a
> virtio-net device with a vhost-user backend. In the case of hardware virtual
> Data Path Acceleration (vDPA) implementation, the hardware configuration, 
> which
> includes tasks like VQ creation and RSS setting, may take above 200ms. This
> significantly increases the downtime of the VM, particularly in terms of
> networking.
>

Hi!

Sorry I totally missed this email. Please CC me in next versions.

Just for completion, there is an ongoing plan to reduce the downtime
in vhost-vdpa. You can find more details at [1].

To send the state periodically is in the roadmap, but some
benchmarking detected that memory pinning and unpinning affects more
to downtime. I'll send a RFC soon with this. The plan was to continue
with iterative state restoring, so I'm happy to know more people are
looking into it!

In the case of vhost-vdpa it already restores the state by not
enabling dataplane until migration completes. All the load is
performed using CVQ, as you can see in
net/vhost-vdpa.c:vhost_vdpa_net_load. After that, all dataplane is
started again.

My idea is to start vhost-vdpa (by calling vhost_vdpa_dev_start) at
the destination at the same moment the migration starts, as it will
not have dataplane enabled. After that, the source should send the
virtio-net vmstate every time it changes. vhost-vdpa net is able to
send and receive through CVQ, so it should be able to modify net
device configuration as many times as needed. I guess that could be
done by calling something in the line of your
vhost_user_set_presetup_state.

This can be improved in vhost-vdpa by being able to send only the new state.

When all the migration is completed, vhost-vdpa net dataplane should
start as it does now.

If you are interested in saving changes to vhost-user protocol, maybe
qemu could just disable the dataplane too with
VHOST_USER_SET_VRING_ENABLE? If not, I think both approaches have a
lot in common, so I'm sure we can develop one backend on top of
another.

Thanks!

[1] https://lists.gnu.org/archive/html/qemu-devel/2023-04/msg00659.html

> To reduce the VM downtime, the proposed approach involves capturing the basic
> device state/configuration during the VM's running stage and performing the
> initial device configuration(presetup). During the normal configuration 
> process
> when the VM is in a stopped state, the second configuration is compared to the
> first one, and only the differences are applied to reduce downtime. Ideally,
> only the vring available index needs to be changed within VM stop.
>
> This feature is disabled by default, because backend like dpdk also needs
> adding support for vhost new message. New device property "x-early-migration"
> can enable this feature.
>
> 1. Register a new vmstate for virtio-net with an early_setup flag to send the
>    device state during migration setup.
> 2. After device state load on destination VM, need to send device status to
>    vhost backend in a new way. Introduce new vhost-user message:
>    VHOST_USER_PRESETUP, to notify backend of presetup.
> 3. Let virtio-net, vhost-net, vhost-dev support presetup. Main flow:
>    a. vhost-dev sending presetup start.
>    b. virtio-net setting mtu.
>    c. vhost-dev sending vring configuration and setting dummy call/kick fd.
>    d. vhost-net sending vring enable.
>    e. vhost-dev sending presetup end.
>
>
> TODOs:
> ======
>   - No vhost-vdpa/kernel support. Need to discuss/design new kernel interface
>     if there's same requirement for vhost-vdpa.
>
>   - No vIOMMU support so far. If there is a need for vIOMMU support, it is
>     planned to be addressed in a follow-up patchset.
>
>
> Test:
> =====
>   - Live migration VM with 2 virtio-net devices, ping can recover.
>     Together with DPDK patch [1].
>   - The time consumption of DPDK function dev_conf is reduced from 191.4 ms
>     to 6.6 ms.
>
>
> References:
> ===========
>
> [1] https://github.com/Mellanox/dpdk-vhost-vfe/pull/37
>
> Any comments or feedback are highly appreciated.
>
> Thanks,
> Yajun
>
>
> Yajun Wu (5):
>   vhost-user: Add presetup protocol feature and op
>   vhost: Add support for presetup
>   vhost-net: Add support for presetup
>   virtio: Add VMState for early load
>   virtio-net: Introduce LM early load
>
>  docs/interop/vhost-user.rst       |  10 ++
>  hw/net/trace-events               |   1 +
>  hw/net/vhost_net.c                |  40 +++++++
>  hw/net/virtio-net.c               | 100 ++++++++++++++++++
>  hw/virtio/vhost-user.c            |  30 ++++++
>  hw/virtio/vhost.c                 | 166 +++++++++++++++++++++++++-----
>  hw/virtio/virtio.c                | 152 ++++++++++++++++-----------
>  include/hw/virtio/vhost-backend.h |   3 +
>  include/hw/virtio/vhost.h         |  12 +++
>  include/hw/virtio/virtio-net.h    |   1 +
>  include/hw/virtio/virtio.h        |  10 +-
>  include/net/vhost_net.h           |   3 +
>  12 files changed, 443 insertions(+), 85 deletions(-)
>
> --
> 2.27.0
>
>