[Qemu-discuss] qemu VM cannot be killed

[Lukáš Kubín]

Hi, we've experienced an issue with kvm instance which got stuck at reboot.
It's an OpenStack environment, with OpenContrail networking (vrouter agent
running on host), Ubuntu 16.04.

Machine was first called to reboot from guest OS by user, had issues with
NFS unmount during that, user sent a hard-reboot call from OpenStack again
then. Then we (platform operator) got involved, tried to "virsh destroy" it
with this output:

error: Failed to destroy domain instance-00004243
error: Failed to terminate process 140529 with SIGKILL: Device or resource
busy


Neither "kill -9" sent to the qemu process helped.

Good guys at OFTC:#virt have guided me to collect the following traces and
ask for help here:

# cat /proc/140529/wchan
vhost_net_ubuf_put_and_wait

# cat /proc/140529/stack
[<ffffffffc03c2f74>] vhost_net_ubuf_put_and_wait+0x54/0xa0 [vhost_net]
[<ffffffffc03c33b4>] vhost_net_ioctl+0x354/0x8a0 [vhost_net]
[<ffffffffa26478c1>] do_vfs_ioctl+0xa1/0x5f0
[<ffffffffa2647e89>] SyS_ioctl+0x79/0x90
[<ffffffffa2c9b7f6>] entry_SYSCALL_64_fastpath+0x1e/0xa8
[<ffffffffffffffff>] 0xffffffffffffffff

The versions we use are:

   - kernel 4.8.0-41-generic
   - qemu-kvm 1:2.5+dfsg-5ubuntu10.2~xenial0+contrail1
   - libvirt-bin 1.3.1-1ubuntu10.1~xenial1+contrail1

What can be the cause for this error? What can we do in such a situation to
destroy the VM - is physical server reboot the only option?

Thanks and greetings,

Lukáš