Re: device compatibility interface for live migration with assigned devices

[Jason Wang]

On 2020/7/16 下午4:32, Yan Zhao wrote:
> On Thu, Jul 16, 2020 at 12:16:26PM +0800, Jason Wang wrote:
> > On 2020/7/14 上午7:29, Yan Zhao wrote:
> > > hi folks,
> > > we are defining a device migration compatibility interface that helps upper
> > > layer stack like openstack/ovirt/libvirt to check if two devices are
> > > live migration compatible.
> > > The "devices" here could be MDEVs, physical devices, or hybrid of the two.
> > > e.g. we could use it to check whether
> > > - a src MDEV can migrate to a target MDEV,
> > > - a src VF in SRIOV can migrate to a target VF in SRIOV,
> > > - a src MDEV can migration to a target VF in SRIOV.
> > >     (e.g. SIOV/SRIOV backward compatibility case)
> > >
> > > The upper layer stack could use this interface as the last step to check
> > > if one device is able to migrate to another device before triggering a real
> > > live migration procedure.
> > > we are not sure if this interface is of value or help to you. please don't
> > > hesitate to drop your valuable comments.
> > >
> > >
> > > (1) interface definition
> > > The interface is defined in below way:
> > >
> > >                __    userspace
> > >                 /\              \
> > >                /                 \write
> > >               / read              \
> > >      ________/__________       ___\|/_____________
> > >     | migration_version |     | migration_version |-->check migration
> > >     ---------------------     ---------------------   compatibility
> > >        device A                    device B
> > >
> > >
> > > a device attribute named migration_version is defined under each device's
> > > sysfs node. e.g. 
> > > (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version).
> >
> > Are you aware of the devlink based device management interface that is
> > proposed upstream? I think it has many advantages over sysfs, do you
> > consider to switch to that?
> not familiar with the devlink. will do some research of it.
> > > userspace tools read the migration_version as a string from the source device,
> > > and write it to the migration_version sysfs attribute in the target device.
> > >
> > > The userspace should treat ANY of below conditions as two devices not 
> > > compatible:
> > > - any one of the two devices does not have a migration_version attribute
> > > - error when reading from migration_version attribute of one device
> > > - error when writing migration_version string of one device to
> > >     migration_version attribute of the other device
> > >
> > > The string read from migration_version attribute is defined by device vendor
> > > driver and is completely opaque to the userspace.
> >
> > My understanding is that something opaque to userspace is not the philosophy
> but the VFIO live migration in itself is essentially a big opaque stream to 
> userspace.


I think it's better not limit to the kernel interface for a specific use 
case. This is basically the device introspection.


> > of Linux. Instead of having a generic API but opaque value, why not do in a
> > vendor specific way like:
> >
> > 1) exposing the device capability in a vendor specific way via sysfs/devlink
> > or other API
> > 2) management read capability in both src and dst and determine whether we
> > can do the migration
> >
> > This is the way we plan to do with vDPA.
> yes, in another reply, Alex proposed to use an interface in json format.
> I guess we can define something like
>
> { "self" :
>    [
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_2",
>        "aggregator"  : "1",
>        "pv-mode"     : "none",
>      }
>    ],
>    "compatible" :
>    [
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_2",
>        "aggregator"  : "1"
>        "pv-mode"     : "none",
>      },
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_4",
>        "aggregator"  : "2"
>        "pv-mode"     : "none",
>      },
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v2",
>        "mdev_type"   : "i915-GVTg_V5_4",
>        "aggregator"  : "2"
>        "pv-mode"     : "none, ppgtt, context",
>      }
>      ...
>    ]
> }


This is probably another call for devlink base interface.


> But as those fields are mostly vendor specific, the userspace can
> only do simple string comparing, I guess the list would be very long as
> it needs to enumerate all possible targets.
> also, in some fileds like "gvt-version", is there a simple way to express
> things like v2+?


That's total vendor specific I think. If "v2+" means it only support a 
version 2+, we can introduce fields like min_version and max_version. 
But again, the point is to let such interfaces vendor specific instead 
of trying to have a generic format.


> If the userspace can read this interface both in src and target and
> check whether both src and target are in corresponding compatible list, I
> think it will work for us.
>
> But still, kernel should not rely on userspace's choice, the opaque
> compatibility string is still required in kernel. No matter whether
> it would be exposed to userspace as an compatibility checking interface,
> vendor driver would keep this part of code and embed the string into the
> migration stream.


Why? Can we simply do:

1) Src support feature A, B, C  (version 1.0)
2) Dst support feature A, B, C, D (version 2.0)
3) only enable feature A, B, C in destination in a version specific way 
(set version to 1.0)
4) migrate metadata A, B, C


>   so exposing it as an interface to be used by libvirt to
> do a safety check before a real live migration is only about enabling
> the kernel part of check to happen ahead.


If we've already exposed the capability, there's no need for an extra 
check like compatibility string.

Thanks


> Thanks
> Yan
>
>
> > > for a Intel vGPU, string format can be defined like
> > > "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator 
> > > count".
> > >
> > > for an NVMe VF connecting to a remote storage. it could be
> > > "PCI ID" + "driver version" + "configured remote storage URL"
> > >
> > > for a QAT VF, it may be
> > > "PCI ID" + "driver version" + "supported encryption set".
> > >
> > > (to avoid namespace confliction from each vendor, we may prefix a driver name to
> > > each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1)
> > >
> > >
> > > (2) backgrounds
> > >
> > > The reason we hope the migration_version string is opaque to the userspace
> > > is that it is hard to generalize standard comparing fields and comparing
> > > methods for different devices from different vendors.
> > > Though userspace now could still do a simple string compare to check if
> > > two devices are compatible, and result should also be right, it's still
> > > too limited as it excludes the possible candidate whose migration_version
> > > string fails to be equal.
> > > e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible
> > > with another MDEV with mdev_type_3, aggregator count 1, even their
> > > migration_version strings are not equal.
> > > (assumed mdev_type_3 is of 3 times equal resources of mdev_type_1).
> > >
> > > besides that, driver version + configured resources are all elements demanding
> > > to take into account.
> > >
> > > So, we hope leaving the freedom to vendor driver and let it make the final 
> > > decision
> > > in a simple reading from source side and writing for test in the target side 
> > > way.
> > >
> > >
> > > we then think the device compatibility issues for live migration with assigned
> > > devices can be divided into two steps:
> > > a. management tools filter out possible migration target devices.
> > >      Tags could be created according to info from product specification.
> > >      we think openstack/ovirt may have vendor proprietary components to create
> > >      those customized tags for each product from each vendor.
> > >      e.g.
> > >      for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to
> > >      search target vGPU are like:
> > >      a tag for compatible parent PCI IDs,
> > >      a tag for a range of gvt driver versions,
> > >      a tag for a range of mdev type + aggregator count
> > >
> > >      for NVMe VF, the tags to search target VF may be like:
> > >      a tag for compatible PCI IDs,
> > >      a tag for a range of driver versions,
> > >      a tag for URL of configured remote storage.
> > >
> > > b. with the output from step a, openstack/ovirt/libvirt could use our proposed
> > >      device migration compatibility interface to make sure the two devices are
> > >      indeed live migration compatible before launching the real live migration
> > >      process to start stream copying, src device stopping and target device
> > >      resuming.
> > >      It is supposed that this step would not bring any performance penalty as
> > >      -in kernel it's just a simple string decoding and comparing
> > >      -in openstack/ovirt, it could be done by extending current function
> > >       check_can_live_migrate_destination, along side claiming target 
> > > resources.[1]
> > >
> > >
> > > [1] 
> > > https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvirt-neutron-sriov-livemigration.html
> > >
> > > Thanks
> > > Yan