Re: [Qemu-discuss] Booting a QEMU instance within a QEMU instance using KVM doesn't work

[Kaushik Datta]

Thank you for the prompt reply, Peter.  Based on your recommendations, I 
downloaded the HEAD of QEMU's master branch a few hours ago and re-ran the test 
below with the "virt" machine model.  After booting, I checked dmesg's KVM 
messages:

[   10.645543] kvm [1]: HYP mode not available

In addition, I see that /dev/kvm doesn't exist.  I also tried the supported 
Xilinx machine models, and the only one that successfully booted and 
initialized KVM's hypervisor was "xlnx-versal-virt".  That's the only reason I 
had used it below- I have no other preference for it.  After booting QEMU with 
xlnx-versal-virt, I again tried setting up the guest QEMU instance.  I didn't 
have time to create a new rootfs with QEMU 4 installed, so for the time being, 
I again used QEMU 3.1.  If I use the "virt" model to boot the guest QEMU, I get 
the following error:

qemu-system-aarch64: kvm_init_vcpu failed: Invalid argument

In fact, I see this error with "xlnx-zcu102" and "xilinx-zynq-a9" as well.  If 
I use "xlnx-versal-virt", then I again see the following error:

... /aarch64-poky-linux/qemu/3.1.0-r0/qemu-3.1.0/exec.c:892: 
cpu_address_space_init: Assertion `asidx == 0 || !kvm_enabled()' failed.

which you already explained below.

Do you happen to know of a specific configuration that properly boots a guest 
QEMU on top of a host QEMU using KVM?  Any other thoughts are welcome as well.

Thanks again,
Kaushik

On 2/5/19, 10:05 AM, "Peter Maydell" <address@hidden> wrote:

    On Tue, 5 Feb 2019 at 13:55, Kaushik Datta <address@hidden> wrote:
    > I am trying to show that KVM is working by booting a guest QEMU instance 
within a host QEMU instance using KVM.  To provide context, the host QEMU is 
translating x86 to aarch64, while the guest QEMU is translating aarch 64 to 
aarch64, so is a candidate for KVM acceleration.  I’m using QEMU 3.1.0 with the 
xlnx-versal-virt machine model for both the host and the guest since this is 
the only QEMU version + machine model combination that I’ve found that 
successfully initializes the KVM hypervisor (shown below).
    
    Personally I would recommend the "virt" board. That's the
    most common one people use to run KVM for aarch64 setups
    (and in general to run Linux in emulation too). I can't
    help so much with the xlnx-versal-virt board, but I would
    suggest not using that unless you specifically need to work
    with an emulation of Xilinx hardware.
    
    Emulation of the virtualization extensions in QEMU should work,
    but it's possible you're running into bugs in it, since it's
    not all that extensively tested.
    
    > In addition, both the host and the guest are using a Debian 4.9.130-2 
kernel and a Yocto-generated rootfs.  Here is the command I use to launch the 
host QEMU instance:
    >
    > qemu-system-aarch64 -M xlnx-versal-virt -m 2048 -cpu cortex-a53 -kernel 
vmlinuz-4.9.0-8-arm64 -initrd new_rootfs.cpio.gz -nographic -D 
"./qemu_debug.txt"
    >
    > After the host QEMU boots, here is the dmesg output regarding KVM:
    >
    > [   12.058077] kvm [1]: HYP VA range: 800000000000:ffffffffffff
    > [   12.061797] kvm [1]: Hyp mode initialized successfully
    >
    > In addition, /dev/kvm exists and I’m operating as root, so that’s good.  
However, when I do lsmod, I get an empty output, which is a concern.
    
    I don't think KVM is a module for aarch64, so this is expected.
    
    >  Anyway, this is the command I used to boot the guest QEMU instance with 
KVM acceleration within the host QEMU:
    >
    > qemu-system-aarch64 -M xlnx-versal-virt -m 1024 -cpu cortex-a53 -kernel 
vmlinuz-4.9.0-8-arm64 -initrd new_rootfs.cpio.gz -accel kvm -enable-kvm 
-nographic -D "./qemu_debug.txt"
    >
    > However, this is the error that I get:
    >
    > qemu-system-aarch64: 
/home/kdatta/hpsc-bsp/HEAD/work/poky_build/tmp/work/aarch64-poky-linux/qemu/3.1.0-r0/qemu-3.1.0/exec.c:892:
 cpu_address_space_init: Assertion `asidx == 0 || !kvm_enabled()' failed.
    
    The error handling here is not great, but the problem I think
    that you're running into is that you're trying to run a setup
    where the guest CPU has the EL3 (TrustZone) feature enabled:
    this can't be emulated with KVM. The xlnx-versal-virt board
    always has this feature enabled, because that's what the Xilinx
    hardware setup is. If you switch to using the "virt" board instead
    (for at least the inner guest, but switching for the outer guest
    too would be sensible) this should go away, since 'virt' defaults
    to not enabling EL3 for the guest CPU.
    
    I would also suggest upgrading to a newer version of QEMU -- the
    command line there suggests you're using 3.1.0-rc0, which is a
    release candidate version of a version of QEMU that is not the
    most recent one. Using an RC for anything other than "testing
    before the release happens" isn't something I'd recommend.
    I also did fix some bugs in emulation of the virtualization
    extensions recently. Using 4.0 should be OK.
    
    thanks
    -- PMM