Re: VFIO Migration

[Yan Zhao]

On Tue, Nov 03, 2020 at 10:13:05AM -0700, Alex Williamson wrote:
> On Tue, 3 Nov 2020 11:03:24 +0000
> Stefan Hajnoczi <stefanha@redhat.com> wrote:

<...>
>  
> > Management tools need to match the device model/configuration from the
> > source device against the destination device. If the destination is
> > capable of supporting the source's device model/configuration then
> > migration can proceed safely.
> > 
> > Let's look at the case where we are migration from an older version of a
> > device to a newer version. On the source we have:
> > 
> >   model = https://vendor-a.com/my-nic
> > 
> > On the destination we have:
> > 
> >   model = https://vendor-a.com/my-nic
> >   rss = on
> > 
> > The two devices are incompatible because the destination exposes the RSS
> > feature that is not present on the source. The RSS feature involves
> > guest-visible hardware interface changes and a change to the device
> > state representation. It is not safe to migrate!
> > 
> > In this case an extra configuration step is necessary so that the
> > destination device can accept the device state from the source. The
> > management tool invokes a vendor-specific tool to put the device into
> > the right configuration:
> > 
> >   # vendor-tool set-migration-config --device 0000:00:04.0 \
> >                                      --model https://vendor-a.com/my-nic
> > 
> > (This tool only succeeds when the device is bound to VFIO but not yet
> > opened.)
> > 
> > The tool invokes ioctls on the vendor-specific VFIO driver that does two
> > things:
> > 1. Tells the device to present the old hardware interface without RSS
> > 2. Uses the old device state representation without RSS support
> > 
> > Does this approach fit?
> 
> 
> Should we not require that any sort of configuration like this occurs
> through sysfs?  We must be able to create an instance with a specific
> configuration without using vendor specific tools, therefore in the
> worse case we should be able to remove and recreate an instance as we
> desire without invoking vendor specific tools.  Thanks,
> 
hi Alex,
could mdevctl serve as a general configuration tool to create/destroy/config
mdev devices?

I think previously the main debate is on what is an easy way for management
tool to find and create a compatible target mdev device according to sysfs
info of source mdev device, is it?
as in [1], we have simplified the method to 1:1 matching of mdev_type
in src and target. and we can further force it to be 1:1 matching of
vendor_specific attributes (e.g. pci id) and dynamic resources
(e.g. aggregator, fps,...) and have mdevctl to create a compatible target
for management tools.

Given management tools like openstack are still in their preliminary
stage of supporting mdev devices, could we first settle down the
compatibility sysfs protocol and treat mdevctl as userspace tool
currently?

[1]: https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg03273.html

Thanks
Yan