Re: [Qemu-discuss] Supported hypervisors running VMs in nested VM

[Rain Maker]

The screenshot and weird behavior I posted is within the L1 VM.

So;
- L0 (host / hypervisor): 0x3a = 5 immediately after boot. L1 VM is
booted WITH -enable-kvm, nested=1, -hypervisor (also tried without
this option), +vmx

- L1 - Linux (VM / "sub-hypervisor"). 0x3a = 0 immediately after boot.
When a L2 VM is booted with -enable-kvm, 0x3a changes to 5 .
- L1 - Windows. 0x3a = ? (most likely, 0) Windows doesn't have tooling
to read MSR as far as I could find.

- L2 (under L1 Linux) - Boots fine. Doesn't matter whether I use
-enable-kvm or not
- L2 (under L1 Windows) - Does not start

As far as I understand it, the BIOS / UEFI should set that MSR to "5".
It should (again, as far as I understand, which is not that much) not
be the task of the operating system.

So, my question is;
- Why would the MSR 0x3a be 0 after boot?
- Why would it change to 5 after starting a L2 VM with -enable-kvm?
- Is it the responsibility of the BIOS / UEFI code to set that MSR (as
it does on my L0 host), or should the OS set this MSR appropriately?

It looks to me like this is a bug somewhere in the Qemu / KVM BIOS
code (MSR returned inappropriately). KVM seems to have a way to
automatically correct this, but Windows does not. I tried this on a VM
booted with the built-in seabios, as well as a VM using the OVMF UEFI
firmware. No difference in behavior.

Thank you very much for the help so far.

Roel Brook




2015-10-07 5:07 GMT+02:00 Bandan Das <address@hidden>:
>
>> On Oct 6, 2015, at 4:43 PM, Rain Maker <address@hidden> wrote:
>>
>> Unfortunately, no difference. WIth or without -hypervisor doesn't make
>> any difference to that flag.
>>
>> But, experimenting on, I found something <very> odd.
>> The 0x3a register is 0 when the VM boots up.
>> Even when I start a L2 VM, 0x3a is still 0.
>>
>> However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
>> this terminal session, which is executed within the L1 VM ("kvmtest").
>> http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
>>
>> I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
>> at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
>> VM as the -enable-kvm VMs do boot.
>>
>> Is this how the MSR is supposed to react?
>>
>> AFAIK, the MSR can only be modified from kernelspace (which also
>> explains why Qemu would only reset it with -enable-kvm, there are no
>> kernelspace components used without it if I understand correctly)
>>
>> Looking at this, I can imagine that Windows does not detect a correct
>> value. It will get 0. Would it make sense to cygwin KVM and see if
>> that changes the MSR register?
>>
>
> You have to use kvm to run (hardware assisted) nested virtualization. I am not
> sure why you think Windows will read 0 for the feature control msr but you 
> have to
> use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When 
> L1 runs L2,
> you don’t have to worry about using —enable-kvm. Hyper-V should automatically 
> detect available
> hardware features available to it and attempt to enable hardware 
> virtualization.
>
>> Sincerely,
>> Roel Brook.
>>
>> 2015-10-05 23:17 GMT+02:00 Bandan Das <address@hidden>:
>>> Rain Maker <address@hidden> writes:
>>>
>>>> Qemu on Linux works fine. I did not even have to explicitly set
>>>> -hypervisor. It simply works.
>>>
>>> Sorry, I meant running Linux with "-hypervisor" to see if specifying
>>> that is somehow messing with the feature flags.
>>>
>>>> As does VirtualBox FYI.
>>>>
>>>> Booting with UEFI didn't make any difference.
>>>>
>>>> After A LOT of Googling, I believe that Hyper-V actually checks bit
>>>> 0x3a of the MSR register (instead of, as the error would .
>>>> This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
>>>> value returned by rdmsr is "0", while on my host it is "5". For
>>>
>>> That seems odd. Even Linux wouldn't work if that value is 0.
>>>
>>>> Hyper-V to work (from what I understand), it should be either 4 or 5.
>>>>
>>>> Googling that, funny enough, brought me back to this list:
>>>> https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
>>>> I guess it really IS important to know what you're googling for to
>>>> find things fast...
>>>>
>>>> That thread simply says that the kernel is too old. Well, my host is
>>>> running 4.2, so should be new enough.
>>>> I'm a bit stuck. Any ideas?
>>>>
>>>> Sincerely,
>>>> Roel Brook
>>>>
>>>>
>>>>
>>>>
>>>> 2015-10-05 21:18 GMT+02:00 Bandan Das <address@hidden>:
>>>>> Rain Maker <address@hidden> writes:
>>>>>
>>>>>> Thanks Bandan.
>>>>>>
>>>>>> That helped a bit. It got me to the next hurdle, as you suspected.
>>>>>>
>>>>>> I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
>>>>>> and the installation now reports "Hyper-V cannot be installed because
>>>>>> virtualization support is not enabled in the BIOS.".
>>>>>
>>>>> Thanks for trying this out.
>>>>>
>>>>>> I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
>>>>>> cannot be installed because virtualization support is not enabled in
>>>>>> the BIOS."
>>>>>>
>>>>>> Apparently, Microsoft queries the BIOS to verify that the
>>>>>
>>>>> When kvm is initialized, it checks for TXT and VMX both being enabled.
>>>>> That too, only if the feature control msr is locked. I don't think there
>>>>> are actually any specific "bios calls" to find this out. I would assume
>>>>> Hyper-V should be doing the same thing but your testing says otherwise.
>>>>>
>>>>> Can you please run linux as L1 with "-hypervisor" and see if it works ?
>>>>> If it doesn't, please check dmesg for relevant messages.
>>>>>
>>>>> Thanks,
>>>>> Bandan
>>>>>
>>>>>
>>>>>> virtualization bit is actually enabled, instead of simply relying on
>>>>>> the VMX flag.
>>>>>> Unfortunately, VMs are still not starting either. The seabios in Qemu
>>>>>> seems to be pretty difficult to modify. I'll check whether I can
>>>>>> reinstall on UEFI, maybe that is going to make a difference.
>>>>>>
>>>>>> The way VMWare does this is actually semi-documented (it hasn't always
>>>>>> been in the product, and a workaround involving manually editing the
>>>>>> configuration has been used for a long time). I'll see if I can
>>>>>> correlate these to Qemu options, to see whether we can use those
>>>>>> instructions to get this working on Qemu.
>>>>>>
>>>>>> 1. Set 'vhv.enable = "TRUE" on the VM
>>>>>>  It "enables virtual hardware virtualization". This seems equivalent
>>>>>> to the -hypervisor flag
>>>>>>
>>>>>> 2. Set 'monitor.virtual_exec = "hardware" on the VM.
>>>>>>  This option seems to force hardware virtualization for both CPU and
>>>>>> MMU. Unsure whether there's an equivalent Qemu configuration option.
>>>>>> Unsure whether it's needed on Qemu. Details at
>>>>>> http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
>>>>>>
>>>>>> 3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
>>>>>>  This seems synonymous to the +vmx flag
>>>>>>
>>>>>> 4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
>>>>>>  I have not found any option to explicitly do this in Qemu. Looking
>>>>>> at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
>>>>>> OK.
>>>>>>
>>>>>> 5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- 
>>>>>> --H- ----
>>>>>>  Not sure how to do that in Qemu or what it does. Looking at
>>>>>> https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
>>>>>> instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
>>>>>> to make any difference whatsoever.
>>>>>>
>>>>>> Sincerely,
>>>>>> Roel Brook
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-10-04 5:07 GMT+02:00 Bandan Das <address@hidden>:
>>>>>>> ...
>>>>>>>> Windows 2012 / 2016 technical preview 3
>>>>>>>> --------------------------------------------------------
>>>>>>>> The installation via the "default" method of Add/Remove Features does
>>>>>>>> not work. Hyper-V displays the error message "A hypervisor is already
>>>>>>>> running".
>>>>>>>>
>>>>>>>> This check can be skipped by using a different method of installation
>>>>>>>> (from PowerShell):
>>>>>>>> Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
>>>>>>>> –All -NoRestart
>>>>>>>>
>>>>>>>> This results in (again) the server booting up, but being unable to run
>>>>>>>> any guest VMs. The error message is less clear then that in 2008, just
>>>>>>>> "The Virtual Machine Management Service failed to start the virtual
>>>>>>>> machine 'New Virtual Machine' because one of the Hyper-V components is
>>>>>>>> not running (Virtual machine ID
>>>>>>>> 0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
>>>>>>>> is what I could find.
>>>>>>>>
>>>>>>>> Other
>>>>>>>> --------
>>>>>>>> Just to verify that "nesting" is actually working, I've also installed
>>>>>>>> a Ubuntu 15.10 VM and installed Qemu on it.
>>>>>>>> This combination CAN successfully run a VM.
>>>>>>>>
>>>>>>>> I've also installed VirtualBox on one of the Windows VMs. This
>>>>>>>> VirtualBox instance is also capable of running virtual machines.
>>>>>>>> According to the icon in the bottom right, VirtualBox IS using the
>>>>>>>> hardware virtualization.
>>>>>>>>
>>>>>>>> Is this a problem specific to Hyper-V? Is there a method to get
>>>>>>>
>>>>>>> Nesting a Hyper-V L1 hypervisor is largely untested. But one of the 
>>>>>>> problems I recollect is that Hyper-V doesn’t like running in a 
>>>>>>> virtualized environment. It checks the “hypervisor” feature flag that 
>>>>>>> Qemu exports. You could try running qemu with “-cpu  host,-hypervisor” 
>>>>>>> or something similar and see if it makes any difference. I suspect 
>>>>>>> there would be other roadblocks though, this is just one of them.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hyper-V working including running guests? I know for a fact that
>>>>>>>> VMWare Workstation / ESX is able to run Hyper-V fully, so it should
>>>>>>>
>>>>>>> Yes, IIRC one of the things ESX does is hide the hypervisor flag 
>>>>>>> specifically for Hyper-V.
>>>>>>>
>>>>>>> Bandan
>>>>>>>
>>>>>>>> not be completely impossible (but I dislike VMWare for different
>>>>>>>> reasons).
>>>>>>>>
>>>>>>>> My Qemu command line (generated by virt-manager). Except for disks and
>>>>>>>> domain names, all are identical:
>>>>>>>>
>>>>>>>> qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
>>>>>>>> pc-i440fx-vivid,accel=kvm,usb=off -cpu
>>>>>>>> SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
>>>>>>>> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
>>>>>>>> 54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
>>>>>>>> -chardev 
>>>>>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
>>>>>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>>>>>>>> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
>>>>>>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>>>>>>>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>>>>>>>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
>>>>>>>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
>>>>>>>> -device 
>>>>>>>> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
>>>>>>>> -device 
>>>>>>>> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
>>>>>>>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
>>>>>>>> file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
>>>>>>>> -device 
>>>>>>>> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>>>>>>>> -drive 
>>>>>>>> file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
>>>>>>>> -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
>>>>>>>> -netdev tap,fd=24,id=hostnet0 -device
>>>>>>>> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
>>>>>>>> -chardev pty,id=charserial0 -device
>>>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev
>>>>>>>> spicevmc,id=charchannel0,name=vdagent -device
>>>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
>>>>>>>> -device usb-tablet,id=input0 -spice
>>>>>>>> port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
>>>>>>>> -device 
>>>>>>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
>>>>>>>> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>>>>>>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
>>>>>>>> spicevmc,id=charredir0,name=usbredir -device
>>>>>>>> usb-redir,chardev=charredir0,id=redir0 -chardev
>>>>>>>> spicevmc,id=charredir1,name=usbredir -device
>>>>>>>> usb-redir,chardev=charredir1,id=redir1 -chardev
>>>>>>>> spicevmc,id=charredir2,name=usbredir -device
>>>>>>>> usb-redir,chardev=charredir2,id=redir2 -chardev
>>>>>>>> spicevmc,id=charredir3,name=usbredir -device
>>>>>>>> usb-redir,chardev=charredir3,id=redir3 -device
>>>>>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>>>>>>>>
>>>>>>>> Thank you in advance for response.
>>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>> Roel Brook
>>>>>>>>
>>>>>>>
>