Re: device compatibility interface for live migration with assigned devices

[Jason Wang]

On 2020/7/20 下午6:39, Sean Mooney wrote:
> On Mon, 2020-07-20 at 11:41 +0800, Jason Wang wrote:
> > On 2020/7/18 上午12:12, Alex Williamson wrote:
> > > On Thu, 16 Jul 2020 16:32:30 +0800
> > > Yan Zhao <yan.y.zhao@intel.com> wrote:
> > >
> > > > On Thu, Jul 16, 2020 at 12:16:26PM +0800, Jason Wang wrote:
> > > > > On 2020/7/14 上午7:29, Yan Zhao wrote:
> > > > > > hi folks,
> > > > > > we are defining a device migration compatibility interface that helps upper
> > > > > > layer stack like openstack/ovirt/libvirt to check if two devices are
> > > > > > live migration compatible.
> > > > > > The "devices" here could be MDEVs, physical devices, or hybrid of the two.
> > > > > > e.g. we could use it to check whether
> > > > > > - a src MDEV can migrate to a target MDEV,
> > > > > > - a src VF in SRIOV can migrate to a target VF in SRIOV,
> > > > > > - a src MDEV can migration to a target VF in SRIOV.
> > > > > >      (e.g. SIOV/SRIOV backward compatibility case)
> > > > > >
> > > > > > The upper layer stack could use this interface as the last step to check
> > > > > > if one device is able to migrate to another device before triggering a real
> > > > > > live migration procedure.
> > > > > > we are not sure if this interface is of value or help to you. please don't
> > > > > > hesitate to drop your valuable comments.
> > > > > >
> > > > > >
> > > > > > (1) interface definition
> > > > > > The interface is defined in below way:
> > > > > >
> > > > > >                 __    userspace
> > > > > >                  /\              \
> > > > > >                 /                 \write
> > > > > >                / read              \
> > > > > >       ________/__________       ___\|/_____________
> > > > > >      | migration_version |     | migration_version |-->check migration
> > > > > >      ---------------------     ---------------------   compatibility
> > > > > >         device A                    device B
> > > > > >
> > > > > >
> > > > > > a device attribute named migration_version is defined under each device's
> > > > > > sysfs node. e.g. 
> > > > > > (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version).
> > > > > Are you aware of the devlink based device management interface that is
> > > > > proposed upstream? I think it has many advantages over sysfs, do you
> > > > > consider to switch to that?
> > > Advantages, such as?
> >
> > My understanding for devlink(netlink) over sysfs (some are mentioned at
> > the time of vDPA sysfs mgmt API discussion) are:
> i tought netlink was used more a as a configuration protocoal to qurry and 
> confire nic and i guess
> other devices in its devlink form requireint a tool to be witten that can speak 
> the protocal to interact with.
> the primary advantate of sysfs is that everything is just a file. there are no 
> addtional depleenceis
> needed


Well, if you try to build logic like introspection on top for a 
sophisticated hardware, you probably need to have library on top. And 
it's attribute per file is pretty inefficient.


>   and unlike netlink there are not interoperatblity issues in a coanitnerised 
> env. if you are using diffrenet
> version of libc and gcc in the contaienr vs the host my understanding is tools 
> like ethtool from ubuntu deployed
> in a container on a centos host can have issue communicating with the host 
> kernel.


Kernel provides stable ABI for userspace, so it's not something that we 
can't fix.


> if its jsut a file unless
> the format the data is returnin in chagnes or the layout of sysfs changes its 
> compatiable regardless of what you
> use to read it.


I believe you can't change sysfs layout which is part of uABI. But as I 
mentioned below, sysfs has several drawbacks. It's not harm to compare 
between different approach when you start a new device management API.

Thanks


> > - existing users (NIC, crypto, SCSI, ib), mature and stable
> > - much better error reporting (ext_ack other than string or errno)
> > - namespace aware
> > - do not couple with kobject
> >
> > Thanks