[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH QEMU v23 08/18] vfio: Register SaveVMHandlers for VFIO device
From: |
Dr. David Alan Gilbert |
Subject: |
Re: [PATCH QEMU v23 08/18] vfio: Register SaveVMHandlers for VFIO device |
Date: |
Thu, 21 May 2020 20:35:38 +0100 |
User-agent: |
Mutt/1.13.4 (2020-02-15) |
* Kirti Wankhede (address@hidden) wrote:
>
>
> On 5/21/2020 7:48 PM, Dr. David Alan Gilbert wrote:
> > * Kirti Wankhede (address@hidden) wrote:
> > > Define flags to be used as delimeter in migration file stream.
> > > Added .save_setup and .save_cleanup functions. Mapped & unmapped migration
> > > region from these functions at source during saving or pre-copy phase.
> > > Set VFIO device state depending on VM's state. During live migration, VM
> > > is
> > > running when .save_setup is called, _SAVING | _RUNNING state is set for
> > > VFIO
> > > device. During save-restore, VM is paused, _SAVING state is set for VFIO
> > > device.
> > >
> > > Signed-off-by: Kirti Wankhede <address@hidden>
> > > Reviewed-by: Neo Jia <address@hidden>
<snip>
> > > + register_savevm_live("vfio", VMSTATE_INSTANCE_ID_ANY, 1,
> > > + &savevm_vfio_handlers, vbasedev);
> >
> > Hi,
> > This is still the only bit which worries me, and I saw your note
> > saying you'd tested it; to calm my nerves, can you run with the
> > 'qemu_loadvm_state_section_startfull' trace enabled with 2 devices
> > and show me the output and qemu command line?
> > I'm trying to figure out how they end up represented in the stream.
> >
>
> Created mtty devices for source VM:
> echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1233" >
> /sys/class/mdev_bus/mtty/mdev_supported_types/mtty-2/create
> echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1234" >
> /sys/class/mdev_bus/mtty/mdev_supported_types/mtty-2/create
>
> for destination VM:
> echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1235" >
> /sys/class/mdev_bus/mtty/mdev_supported_types/mtty-2/create
> echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1236" >
> /sys/class/mdev_bus/mtty/mdev_supported_types/mtty-2/create
>
> Source qemu-cmdline:
> /usr/libexec/qemu-kvm \
> -name guest=rhel75-mig,debug-threads=on \
> -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off \
> -cpu SandyBridge,vme=on,hypervisor=on,arat=on,xsaveopt=on \
> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 \
> -uuid eefb718c-137c-d416-e573-dd74ecd3490d \
> -drive
> file=/home/vm/rhel-75.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,cache=none
> \
> -device
> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1,write-cache=on
> \
> -vnc 127.0.0.1:0 \
> -device rtl8139,netdev=net0,mac=52:54:b2:88:86:2a,bus=pci.0,addr=0x3
> -netdev tap,id=net0,script=/root/qemu-ifup,downscript=no \
> -device
> vfio-pci,sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1233
> \
> -device
> vfio-pci,sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1234
> \
> --trace events=/root/vfio_events \
> -monitor unix:/tmp/qmp_socket1,server,nowait \
> -serial stdio \
> -msg timestamp=on
>
> Destination qemu-cmdline:
> /usr/libexec/qemu-kvm \
> -name guest=rhel75-mig,debug-threads=on \
> -machine pc-i440fx-3.1,accel=kvm,usb=off,dump-guest-core=off \
> -cpu SandyBridge,vme=on,hypervisor=on,arat=on,xsaveopt=on \
> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 \
> -uuid eefb718c-137c-d416-e573-dd74ecd3490d \
> -drive
> file=/home/vm/rhel-75.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,cache=none
> \
> -device
> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1,write-cache=on
> \
> -vnc 127.0.0.1:1 \
> -device rtl8139,netdev=net0,mac=52:54:b2:88:86:2a,bus=pci.0,addr=0x3
> -netdev tap,id=net0,script=/root/qemu-ifup,downscript=no \
> -device
> vfio-pci,sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1235
> \
> -device
> vfio-pci,sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1236
> \
> -incoming unix:/tmp/mig_socket \
> --trace events=/root/vfio_events \
> -monitor unix:/tmp/qmp_socket2,server,nowait \
> -serial stdio \
> -msg timestamp=on
>
> Migrate:
> echo "migrate_set_speed 0" | sudo nc -U /tmp/qmp_socket1
> echo "migrate -d unix:/tmp/mig_socket" | sudo nc -U $/tmp/qmp_socket1
>
> After migration, 'qemu_loadvm_state_section_startfull' traces:
>
> qemu_loadvm_state_section_startfull 0.000 pid=1457 section_id=0x2
> idstr=b'ram' instance_id=0x0 version_id=0x4
> qemu_loadvm_state_section_startfull 515.606 pid=1457 section_id=0x2e
> idstr=b'vfio' instance_id=0x0 version_id=0x1
> qemu_loadvm_state_section_startfull 10.661 pid=1457 section_id=0x2f
> idstr=b'vfio' instance_id=0x1 version_id=0x1
Right, so this is my worry - we have two devices in the stream called
'vfio' with I think sequential id's - what makes each of your source
vfio devices go to the correct destination vfio device? If the two
devices were different vfio devices, how would you ensure that they
ended up in the write place? There's no requirement for
the order of the qemu command line on the source and the destination
to be the same, or for qemu to maintain semantics based on the order -
but I bet that's the ordering were getting here.
> idstr=b'0000:00:03.0/rtl8139' instance_id=0x0 version_id=0x5
Now you see that PCI NIC has a nice PCI address as it's name in the
stream; if you have two NICs defined then they end up getting loaded
into the destination device with the same guest PCI address - so it's
nice and repeatable (especially if you specify the PCI devices address
on the command line).
Dave
> qemu_loadvm_state_section_startfull 1619.840 pid=1457 section_id=0x30
> idstr=b'acpi_build' instance_id=0x0 version_id=0x1
> qemu_loadvm_state_section_startfull 4.200 pid=1457 section_id=0x31
> idstr=b'globalstate' instance_id=0x0 version_id=0x1
>
> Thanks,
> Kirti
>
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK
- Re: [PATCH QEMU v23 04/18] vfio: Add save and load functions for VFIO PCI devices, (continued)
- [PATCH QEMU v23 09/18] vfio: Add save state functions to SaveVMHandlers, Kirti Wankhede, 2020/05/20
- [PATCH QEMU v23 10/18] vfio: Add load state functions to SaveVMHandlers, Kirti Wankhede, 2020/05/20
- [PATCH QEMU v23 11/18] iommu: add callback to get address limit IOMMU supports, Kirti Wankhede, 2020/05/20
- [PATCH QEMU v23 12/18] memory: Set DIRTY_MEMORY_MIGRATION when IOMMU is enabled, Kirti Wankhede, 2020/05/20
- [PATCH QEMU v23 13/18] vfio: Get migration capability flags for container, Kirti Wankhede, 2020/05/20