Re: [Qemu-devel] Logging dirty pages from vhost-net in-kernel with vIOMMU

[Jintack Lim]

On Fri, Dec 7, 2018 at 7:37 AM Jason Wang <address@hidden> wrote:
>
>
> On 2018/12/6 下午8:44, Jason Wang wrote:
> >
> > On 2018/12/6 下午8:11, Jintack Lim wrote:
> >> On Thu, Dec 6, 2018 at 2:33 AM Jason Wang <address@hidden> wrote:
> >>>
> >>> On 2018/12/5 下午10:47, Jintack Lim wrote:
> >>>> On Tue, Dec 4, 2018 at 8:30 PM Jason Wang <address@hidden> wrote:
> >>>>> On 2018/12/5 上午2:37, Jintack Lim wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'm wondering how the current implementation works when logging
> >>>>>> dirty
> >>>>>> pages during migration from vhost-net (in kernel) when used vIOMMU.
> >>>>>>
> >>>>>> I understand how vhost-net logs GPAs when not using vIOMMU. But when
> >>>>>> we use vhost with vIOMMU, then shouldn't vhost-net need to log the
> >>>>>> translated address (GPA) instead of the address written in the
> >>>>>> descriptor (IOVA) ? The current implementation looks like vhost-net
> >>>>>> just logs IOVA without translation in vhost_get_vq_desc() in
> >>>>>> drivers/vhost/net.c. It seems like QEMU doesn't do any further
> >>>>>> translation of the dirty log when syncing.
> >>>>>>
> >>>>>> I might be missing something. Could somebody shed some light on
> >>>>>> this?
> >>>>> Good catch. It looks like a bug to me. Want to post a patch for this?
> >>>> Thanks for the confirmation.
> >>>>
> >>>> What would be a good setup to catch this kind of migration bug? I
> >>>> tried to observe it in the VM expecting to see network applications
> >>>> not getting data correctly on the destination, but it was not
> >>>> successful (i.e. the VM on the destination just worked fine.) I didn't
> >>>> even see anything going wrong when I disabled the vhost logging
> >>>> completely without using vIOMMU.
> >>>>
> >>>> What I did is I ran multiple network benchmarks (e.g. netperf tcp
> >>>> stream and my own one to check correctness of received data) in a VM
> >>>> without vhost dirty page logging, and the benchmarks just ran fine in
> >>>> the destination. I checked the used ring at the time the VM is stopped
> >>>> in the source for migration, and it had multiple descriptors that is
> >>>> (probably) not processed in the VM yet. Do you have any insight how it
> >>>> could just work and what would be a good setup to catch this?
> >>>
> >>> According to past experience, it could be reproduced by doing scp from
> >>> host to guest during migration.
> >>>
> >> Thanks. I actually tried that, but didn't see any problem either - I
> >> copied a large file during migration from host to guest (the copy
> >> continued on the destination), and checked md5 hashes using md5sum,
> >> but the copied file had the same checksum as the one in the host.
> >>
> >> Do you recall what kind of symptom you observed when the dirty pages
> >> were not migrated correctly with scp?
> >
> >
> > Yes,  the point is to make the migration converge before the end of
> > scp (e.g set migration speed to a very big value). If scp end before
> > migration, we won't catch the bug. And it's better to do several
> > rounds of migration during scp.
> >
> > Anyway, let me try to reproduce it tomorrow.
> >
>
> Looks like I can reproduce this, scp give the following error to me:
>
> scp /home/file address@hidden:/home
> file                                           63% 1301MB 58.1MB/s
> 00:12 ETAReceived disconnect from 192.168.100.4: 2: Packet corrupt
> lost connection

Thanks for sharing this.

I was able to reproduce the bug. I observed different md5sum in the
host and the guest after several tries. I didn't observe the
disconnect you saw, but the different md5sum is enough to show the
bug, I guess.

Thanks,
Jintack

>
> FYI, I use the following cli:
>
> numactl --cpunodebind 0 --membind 0 $qemu_path $img_path \
>             -netdev tap,id=hn0,vhost=on \
>             -device ioh3420,id=root.1,chassis=1 \
>             -device
> virtio-net-pci,bus=root.1,netdev=hn0,ats=on,disable-legacy=on,disable-modern=off,iommu_platform=on
> \
>             -device intel-iommu,device-iotlb=on \
>             -M q35 -m 4G -enable-kvm -cpu host -smp 2 $@
>
> Thanks
>
>
> > Thanks
>