[Qemu-discuss] Qemu freeze on I/O intensive workload

[Jean-Tiare LE BIGOT]

Hi,

I am using Qemu in a test suite to virtualize 3 x86_64 machines in an
isolated network. The end goal is to run integration tests on a Yocto
generated distribution.

In the setupe phase of the test suite, we start 4 Qemu instances in
parallel with the "-daemonize" option and a global lock to prevent parralel
start (until daemonized).

The machine's configuration files start as follow (prior to variable
replacement):

>
> #
> # General setup
> #
>
> [machine]
>   accel = "kvm" # For acceleration
>   type = "q35"  # For AHCI drive (sda instead of ide)
>
> [memory]
>   size = "1G"
>
> #
> # Control socket
> #
>
> [chardev "monitor"]
>   backend = "socket"
>   server = "on"
>   wait = "off"
>   path = "%%MACHINE_DATA%%/monitor.sock"
>
> [mon]
>   chardev = "monitor"
>   mode = "control"
>
> #
> # UEFI setup
> #
>
> [drive]
>   if = "pflash"
>   format = "raw"
>   readonly = "on"
>   file = "%%MACHINE_DATA%%/OVMF_CODE.fd"
>
> [drive]
>   if = "pflash"
>   format = "raw"
>   file = "%%MACHINE_DATA%%/OVMF_VARS.fd"
>
> #
> # Harddrive and install ISO
> #
>
> [drive]
>   if = "ide"
>   format = "raw"
>   file = "%%MACHINE_DATA%%/sda.img"
>
> [drive]
>   if = "ide"
>   index = "2"
>   format = "raw"
>   file = "%%MACHINE_DATA%%/cdrom-rw.iso"
>

When started, the machines install themeselves. The install is based on a
"dd" of a pre-generated ext4 image to the new partitions set.

What we observe is, sometime, the virtualized machine freezes in the "dd"
step. From the host side, we observe that the "voluntary_ctxt_switches" of
the Qemu thread does not increase in 3 seconds suggesting that the emulated
process is blocked.

The stack trace (/proc/[Thread ID]/stack) is:

[<ffffffffc055565a>] kvm_vcpu_block+0x8a/0x2f0 [kvm]
> [<ffffffffc0571529>] kvm_arch_vcpu_ioctl_run+0x159/0x1620 [kvm]
> [<ffffffffc0555146>] kvm_vcpu_ioctl+0x2a6/0x620 [kvm]
> [<ffffffffaf287b55>] do_vfs_ioctl+0xa5/0x600
> [<ffffffffaf288129>] SyS_ioctl+0x79/0x90
> [<ffffffffaf89c0b7>] entry_SYSCALL_64_fastpath+0x1a/0xa5
> [<ffffffffffffffff>] 0xffffffffffffffff
>

We are using Qemu 3.0 with a kernel 4.13.9-300.fc27.x86_64 on the host. The
guest is a Yocto Linux 4.9 with a custom stripped down configuration.

We do not know where the freeze comes from. From what we observe, the
freeze may come from the host kernel, the guest kernel or Qemu itself.

How can we go further in the diagnose ? We can enable traces / patch / run
under gdb / ...

Thanks !

-- 
Jean-Tiare Le Bigot