[Qemu-discuss] Guest crash shortly after restoring a snapshot of a running VM with nested Xen inside

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello all,

I'm trying to use snapshots on a KVM with enabled nested virtualization
and Xen running inside. Generally nested virtualization works pretty
well here. Until I try to restore a live snapshot. After doing so, the
VM crashes with (serial output from Xen below). No specific output on
qemu stderr. Any ideas? Is there some option I can enable to get more
info from qemu?

Xen output (similar for all its CPUs):

    (XEN) ----[ Xen-4.8.5-5.fc25  x86_64  debug=n   Not tainted ]----
    (XEN) CPU:    1
    (XEN) RIP:    e008:[<ffff82d0801e2329>] vmcs.c#vmx_load_vmcs+0x59/0xa0
    (XEN) RFLAGS: 0000000000010013   CONTEXT: hypervisor (d3v0)
    (XEN) rax: 0000000000000286   rbx: ffff8300849ab000   rcx: 0000000000000000
    (XEN) rdx: 00000000817fc000   rsi: ffff8301bc017da8   rdi: ffff8300849ab000
    (XEN) rbp: ffff8300bfc7d000   rsp: ffff8301bc017dc0   r8:  0000000000000000
    (XEN) r9:  0000000000000000   r10: ffff830083ab1000   r11: ffff82e000978100
    (XEN) r12: ffff830083ab1000   r13: ffff8300bfc93000   r14: 0000000000000001
    (XEN) r15: ffff82d0802d4720   cr0: 0000000080050033   cr4: 00000000003526e0
    (XEN) cr3: 0000000082255000   cr2: 0000725a16f80670
    (XEN) fsb: 0000000000000000   gsb: ffff888018400000   gss: 0000000000000000
    (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
    (XEN) Xen code around <ffff82d0801e2329> (vmcs.c#vmx_load_vmcs+0x59/0xa0):
    (XEN)  05 00 00 48 89 54 24 08 <0f> c7 74 24 08 0f 86 3e 38 00 00 48 89 e1 
48 8b
    (XEN) Xen stack trace from rsp=ffff8301bc017dc0:
    (XEN)    ffff830083ab1000 00000000817fc000 ffff82d0801e506a ffff8300849ab000
    (XEN)    ffff82d08015ba95 0000000000000000 0000000000000000 0000000000000000
    (XEN)    0000000000000000 ffff82d08021242e ffff8300849ab000 ffff8300bfc7d000
    (XEN)    0000000000000002 ffff8301bc01c148 00000046c9e3df7b ffff82d0802d4720
    (XEN)    ffff82d080126e40 0000000001c9c380 ffff8301bc01c160 ffff82d08021242e
    (XEN)    ffff8301bc017ef8 0000000000000001 ffff8300849ab000 0000000001c9c380
    (XEN)    ffff82d080212400 ffff82d08021242e ffffffffffffffff ffff8301bc017fff
    (XEN)    ffff82d0802c2900 ffff82d0802ca900 0000000000000001 0000000000000000
    (XEN)    ffff82d08012a262 ffff8300bfc7d000 0000000000000000 0000000000000000
    (XEN)    0000000000000000 0000000000000000 ffff82d08020f876 0000000000000000
    (XEN)    0000000000000000 ffffffff82417740 ffffffff82417740 0000000000000000
    (XEN)    0000000000000000 0000000000000000 ffffc90000693d78 0000000000000000
    (XEN)    ffffc9000092be90 ffffffff8195c310 0000000000000000 0000000000000000
    (XEN)    0000000000000087 0000000000000000 000000fa00000000 ffffffff8195c6d2
    (XEN)    0000000000000000 0000000000000246 ffffffff82403e70 0000000000000000
    (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
    (XEN)    0000000000000001 ffff8300849ab000 000000313bd34980 00000000003526e0
    (XEN)    0000000000000000 0000000000000000 0000060100000000 0000000000000000
    (XEN) Xen call trace:
    (XEN)    [<ffff82d0801e2329>] vmcs.c#vmx_load_vmcs+0x59/0xa0
    (XEN)    [<ffff82d0801e506a>] vmx_do_resume+0xfa/0x110
    (XEN)    [<ffff82d08015ba95>] context_switch+0x1a5/0xf40
    (XEN)    [<ffff82d08021242e>] lstar_enter+0xae/0x120
    (XEN)    [<ffff82d080126e40>] schedule.c#schedule+0x230/0x5b0
    (XEN)    [<ffff82d08021242e>] lstar_enter+0xae/0x120
    (XEN)    [<ffff82d080212400>] lstar_enter+0x80/0x120
    (XEN)    [<ffff82d08021242e>] lstar_enter+0xae/0x120
    (XEN)    [<ffff82d08012a262>] softirq.c#__do_softirq+0x62/0x90
    (XEN)    [<ffff82d08020f876>] entry.o#process_softirqs+0x6/0x20
    (XEN) 
    (XEN) 
    (XEN) ****************************************
    (XEN) Panic on CPU 1:
    (XEN) FATAL TRAP: vector = 6 (invalid opcode)
    (XEN) ****************************************

QEMU version:

    QEMU emulator version 3.0.0 (qemu-3.0.0-4.fc29)

QEMU command line (initial):

/usr/bin/qemu-system-x86_64 \
    -machine accel=kvm \
    -vga cirrus \
    -only-migratable \
    -chardev ringbuf,id=serial0,logfile=serial0,logappend=on \
    -serial chardev:serial0 \
    -soundhw ac97 \
    -global isa-fdc.driveA= \
    -m 6144 \
    -cpu host,+vmx \
    -netdev user,id=qanet0 \
    -device e1000e,netdev=qanet0,mac=52:54:00:12:34:56 \
    -boot once=d,menu=on,splash-time=5000 \
    -device usb-ehci \
    -device usb-tablet \
    -smp 2 \
    -enable-kvm \
    -no-shutdown \
    -vnc :91,share=force-shared \
    -device virtio-serial \
    -chardev 
socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log,logappend=on
 \
    -device 
virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console \
    -chardev 
socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on
 \
    -qmp chardev:qmp_socket \
    -S \
    -device virtio-scsi-pci,id=scsi0 \
    -blockdev 
driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/1/raid/hd0-overlay0,cache.no-flush=on
 \
    -blockdev 
driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on \
    -device scsi-hd,id=hd0-device,drive=hd0-overlay0,serial=hd0

QEMU command line (load snapshot):

/usr/bin/qemu-system-x86_64 \
    -vga cirrus \
    -only-migratable \
    -chardev ringbuf,id=serial0,logfile=serial0,logappend=on \
    -serial chardev:serial0 \
    -soundhw ac97 \
    -global isa-fdc.driveA= \
    -m 6144 \
    -cpu host,+vmx \
    -netdev user,id=qanet0 \
    -device e1000e,netdev=qanet0,mac=52:54:00:12:34:56 \
    -boot once=d,menu=on,splash-time=5000 \
    -device usb-ehci \
    -device usb-tablet \
    -smp 2 \
    -enable-kvm \
    -no-shutdown \
    -vnc :91,share=force-shared \
    -device virtio-serial \
    -chardev 
socket,path=virtio_console,server,nowait,id=virtio_console,logfile=virtio_console.log,logappend=on
 \
    -device 
virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console \
    -chardev 
socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on
 \
    -qmp chardev:qmp_socket \
    -S \
    -device virtio-scsi-pci,id=scsi0 \
    -blockdev 
driver=file,node-name=hd0-overlay1-file,filename=/var/lib/openqa/pool/1/raid/hd0-overlay1,cache.no-flush=on
 \
    -blockdev 
driver=qcow2,node-name=hd0-overlay1,file=hd0-overlay1-file,cache.no-flush=on \
    -device scsi-hd,id=hd0-device,drive=hd0-overlay1,serial=hd0 \
    -incoming defer

- -- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlyk45UACgkQ24/THMrX
1yw73ggAgw1VIhgmG5+REqnHrL65czVGFnj6qpLP75NzPvmLYfp6EUZa4lM6p3bD
9gu/0+k5xox5D1yICdgquT0xbWuF6dqDhQKnIzC8rVAF1O1wrSxX3t20LwYaBUg9
geL9BwBkad1bwfinzpgzc95qwGDxRMoBNTovsfvh8KwkH516qfYkMKwQ7N4ppkYi
Cf1nkpIsANog4/WG12soZSML1UeJq1PfZ3OlqEYREdBC3zz5WJjDddWIw12ss8KL
8M7UbpZYdwc6S+d2ySR20NGNdwJc7HNJnNGByXHJ427FUAp+nb7EoAQ/cYXvod9G
qcl1YzKHkZ4pgRisJXv+74FLhfr5wQ==
=mD60
-----END PGP SIGNATURE-----