Re: [RFC PATCH 0/3] vfio/migration: Support manual clear vfio dirty log

[Kunkun Jiang]

Hi Kevin,

On 2021/3/18 14:28, Tian, Kevin wrote:
> > From: Kunkun Jiang
> > Sent: Wednesday, March 10, 2021 5:41 PM
> >
> > Hi all,
> >
> > In the past, we clear dirty log immediately after sync dirty log to
> > userspace. This may cause redundant dirty handling if userspace
> > handles dirty log iteratively:
> >
> > After vfio clears dirty log, new dirty log starts to generate. These
> > new dirty log will be reported to userspace even if they are generated
> > before userspace handles the same dirty page.
> >
> > Since a new dirty log tracking method for vfio based on iommu hwdbm[1]
> > has been introduced in the kernel and added a new capability named
> > VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty
> > handling by supporting it.
> Is there any performance data showing the benefit of this new method?
Current dirty log tracking method for VFIO:
[1] All pages marked dirty if not all iommu_groups have pinned_scope
[2] pinned pages by various vendor drivers if all iommu_groups have 
pinned scope

Both methods are coarse-grained and can not determine which pages are
really dirty. Each round may mark the pages that are not really dirty as 
dirty
and send them to the destination. ( It might be better if the range of the
pinned_scope was smaller. ) This will result in a waste of resources.

HWDBM is short for Hardware Dirty Bit Management.
(e.g. smmuv3 HTTU, Hardware Translation Table Update)

About SMMU HTTU:
HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty
state of the TTD (Translation Table Descriptor) by hardware.

With HTTU, stage1 TTD is classified into 3 types:
                                 DBM bit AP[2](readonly bit)
1. writable_clean          1                            1
2. writable_dirty           1                            0
3. readonly                   0                            1

If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from
writable_clean to writable_dirty. Then software can scan TTD to sync dirty
state into dirty bitmap. With this feature, we can track the dirty log of
DMA continuously and precisely.

The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on
the KVM side. We add this new log_clear() interface only to split the old
log_sync() into two separated procedures:

- use log_sync() to collect the collection only, and,
- use log_clear() to clear the dirty bitmap.

If you're interested in this new method, you can take a look at our set of
patches.
[1] 
https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/

Best regards,
Kunkun Jiang

> > This series include patches as below:
> > Patch 1:
> > - updated the linux-headers/linux/vfio.h from kernel side
> >
> > Patch 2:
> > - introduced 'struct VFIODMARange' to describe a range of the given DMA
> >    mapping and with respect to a VFIO_IOMMU_MAP_DMA operation
> >
> > Patch 3:
> > - implemented the operation to manual clear vfio dirty log, which can
> >    eliminate some redundant dirty handling
> >
> > Thanks,
> > Kunkun Jiang
> >
> > [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-
> > zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f
> > 863
> >
> > Zenghui Yu (3):
> >    linux-headers: update against 5.12-rc2 and "vfio log clear" series
> >    vfio: Maintain DMA mapping range for the container
> >    vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP
> >
> >   hw/vfio/common.c              | 207 ++++++++++++++++++++++++++++++++--
> >   include/hw/vfio/vfio-common.h |  10 ++
> >   linux-headers/linux/vfio.h    |  55 ++++++++-
> >   3 files changed, 264 insertions(+), 8 deletions(-)
> >
> > --
> > 2.23.0
>
> .