Re: [Qemu-devel] vhost, iova, and dirty page tracking

[Jason Wang]

On 2019/9/17 下午4:48, Tian, Kevin wrote:
> > From: Jason Wang [mailto:address@hidden]
> > Sent: Monday, September 16, 2019 4:33 PM
> >
> >
> > On 2019/9/16 上午9:51, Tian, Kevin wrote:
> > > Hi, Jason
> > >
> > > We had a discussion about dirty page tracking in VFIO, when vIOMMU
> > > is enabled:
> > >
> > > https://lists.nongnu.org/archive/html/qemu-devel/2019-
> > 09/msg02690.html
> > > It's actually a similar model as vhost - Qemu cannot interpose the fast-
> > path
> > > DMAs thus relies on the kernel part to track and report dirty page
> > information.
> > > Currently Qemu tracks dirty pages in GFN level, thus demanding a
> > translation
> > > from IOVA to GPA. Then the open in our discussion is where this
> > translation
> > > should happen. Doing the translation in kernel implies a device iotlb
> > flavor,
> > > which is what vhost implements today. It requires potentially large
> > tracking
> > > structures in the host kernel, but leveraging the existing log_sync flow in
> > Qemu.
> > > On the other hand, Qemu may perform log_sync for every removal of
> > IOVA
> > > mapping and then do the translation itself, then avoiding the GPA
> > awareness
> > > in the kernel side. It needs some change to current Qemu log-sync flow,
> > and
> > > may bring more overhead if IOVA is frequently unmapped.
> > >
> > > So we'd like to hear about your opinions, especially about how you came
> > > down to the current iotlb approach for vhost.
> >
> > We don't consider too much in the point when introducing vhost. And
> > before IOTLB, vhost has already know GPA through its mem table
> > (GPA->HVA). So it's nature and easier to track dirty pages at GPA level
> > then it won't any changes in the existing ABI.
> This is the same situation as VFIO.
>
> > For VFIO case, the only advantages of using GPA is that the log can then
> > be shared among all the devices that belongs to the VM. Otherwise
> > syncing through IOVA is cleaner.
> I still worry about the potential performance impact with this approach.
> In current mdev live migration series, there are multiple system calls
> involved when retrieving the dirty bitmap information for a given memory
> range.


I haven't took a deep look at that series. Technically dirty bitmap 
could be shared between device and driver, then there's no system call 
in synchronization.


> IOVA mappings might be changed frequently. Though one may
> argue that frequent IOVA change already has bad performance, it's still
> not good to introduce further non-negligible overhead in such situation.


Yes, it depends on the behavior of vIOMMU driver, e.g the frequency and 
granularity of the flushing.


> On the other hand, I realized that adding IOVA awareness in VFIO is
> actually easy. Today VFIO already maintains a full list of IOVA and its
> associated HVA in vfio_dma structure, according to VFIO_MAP and
> VFIO_UNMAP. As long as we allow the latter two operations to accept
> another parameter (GPA), IOVA->GPA mapping can be naturally cached
> in existing vfio_dma objects.


Note that the HVA to GPA mapping is not an 1:1 mapping. One HVA range 
could be mapped to several GPA ranges.


>   Those objects are always updated according
> to MAP and UNMAP ioctls to be up-to-date. Qemu then uniformly
> retrieves the VFIO dirty bitmap for the entire GPA range in every pre-copy
> round, regardless of whether vIOMMU is enabled. There is no need of
> another IOTLB implementation, with the main ask on a v2 MAP/UNMAP
> interface.


Or provide GPA to HVA mapping as vhost did. But a question is, I believe 
device can only do dirty page logging through IOVA. So how do you handle 
the case when IOVA is removed in this case?

Thanks


> Alex, your thoughts?
>
> Thanks
> Kevin