RE: [RFC PATCH 0/3] vfio/migration: Support manual clear vfio dirty log

[Tian, Kevin]

> From: Kunkun Jiang <jiangkunkun@huawei.com>
> Sent: Thursday, March 18, 2021 3:59 PM
> 
> Hi Kevin,
> 
> On 2021/3/18 14:28, Tian, Kevin wrote:
> >> From: Kunkun Jiang
> >> Sent: Wednesday, March 10, 2021 5:41 PM
> >>
> >> Hi all,
> >>
> >> In the past, we clear dirty log immediately after sync dirty log to
> >> userspace. This may cause redundant dirty handling if userspace
> >> handles dirty log iteratively:
> >>
> >> After vfio clears dirty log, new dirty log starts to generate. These
> >> new dirty log will be reported to userspace even if they are generated
> >> before userspace handles the same dirty page.
> >>
> >> Since a new dirty log tracking method for vfio based on iommu hwdbm[1]
> >> has been introduced in the kernel and added a new capability named
> >> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant
> dirty
> >> handling by supporting it.
> > Is there any performance data showing the benefit of this new method?
> >
> Current dirty log tracking method for VFIO:
> [1] All pages marked dirty if not all iommu_groups have pinned_scope
> [2] pinned pages by various vendor drivers if all iommu_groups have
> pinned scope
> 
> Both methods are coarse-grained and can not determine which pages are
> really dirty. Each round may mark the pages that are not really dirty as
> dirty
> and send them to the destination. ( It might be better if the range of the
> pinned_scope was smaller. ) This will result in a waste of resources.
> 
> HWDBM is short for Hardware Dirty Bit Management.
> (e.g. smmuv3 HTTU, Hardware Translation Table Update)
> 
> About SMMU HTTU:
> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty
> state of the TTD (Translation Table Descriptor) by hardware.
> 
> With HTTU, stage1 TTD is classified into 3 types:
>                                   DBM bit AP[2](readonly bit)
> 1. writable_clean          1                            1
> 2. writable_dirty           1                            0
> 3. readonly                   0                            1
> 
> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from
> writable_clean to writable_dirty. Then software can scan TTD to sync dirty
> state into dirty bitmap. With this feature, we can track the dirty log of
> DMA continuously and precisely.
> 
> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on
> the KVM side. We add this new log_clear() interface only to split the old
> log_sync() into two separated procedures:
> 
> - use log_sync() to collect the collection only, and,
> - use log_clear() to clear the dirty bitmap.
> 
> If you're interested in this new method, you can take a look at our set of
> patches.
> [1]
> https://lore.kernel.org/linux-iommu/20210310090614.26668-1-
> zhukeqian1@huawei.com/
> 

I know what you are doing. Intel is also working on VT-d dirty bit support
based on above link. What I'm curious is the actual performance gain 
with this optimization. KVM doing that is one good reference, but IOMMU
has different characteristics (e.g. longer invalidation latency) compared to 
CPU MMU. It's always good to understand what a so-called optimization
can actually optimize in a context different from where it's originally proved.😊

Thanks
Kevin