Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?

[Xu Liu]

Hello,

Recently I am running my programs in QEMU (x86_64) with “-accel=kvm”.  The QEMU version is 6.0.0.

I run my programs in two ways:

1.   I pass through my device through vfio-pci  to QEMU,  this way works well.

2.  I write an emulated PCI device for QEMU, and run my programs on the emulated PCI device.  

     This crashes when  the code try to do memory copy to PCI device when the data length is longer than 16 bytes.

     While the  passthrough device works well for the same situation.

After  dump the assembly code.  I noticed when the data is <= 16 bytes,  the mov assembly code is chosen, and it works well.

When the data is > 16 bytes,  the vmovdqu  assembly code is chosen, and it crashes with “illegal operand”.

Given the code and data are exactly same for both passthrough device and emulated device.  I am curious about why this happens.

After turn on  kernel trace for kvm by   echo kvm:*   /sys/kernel/debug/tracing/set_event

And rerun the QEMU and my code for both passthrough device and emulated device, I noticed that:

1) for passthrough device,  I didn’t see  any trace events related to my  gva and gpa.  This makes me think that the memory copy to PCI device went through different code path . It is handled by the guest OS without exit to VMX.

2) for emulated device, if I use   compiler flag  target-feature=-avx,-avx2 to force compiler use  mov assembly code,  I can see the memory copy goes through the KVM_EXIT_MMIO, and everything works well.

    if I don’t force the compiler use mov ,  the compiler just chooses the vmovdqu ,  which just crash the programs, and no KVM_EXIT_MMIO related to my memory copy appears in the trace events. Looks like the guest OS handles the crash.

Any clue about why the vmovdqu works for passthrough device but not work for emulated device.

Thanks