Re: More than 255 vcpus Windows VM setup without viommu ?

[Sandesh Patel]

> On 11 Jul 2024, at 12:56 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> 
> On Mon, 2024-07-08 at 10:13 +0100, David Woodhouse wrote:
>> On Wed, 2024-07-03 at 16:01 +0000, Sandesh Patel wrote:
>>>> 
>>>> Interesting. What exactly has Windows *done* in those MSI entries?
>>>> That might give a clue about how to support it.
> 
> I repeated my experiment with interrupt-only remapping (no DMA
> remapping). On two hosts:
> 
> 
> 
> vendor_id       : AuthenticAMD
> cpu family      : 25
> model           : 17
> model name      : AMD EPYC 9R14 96-Core Processor
> 
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 143
> model name      : Intel(R) Xeon(R) Platinum 8488C
> 
> I used identical command lines on both, and on each host I got the same
> result with all of '-cpu host', '-cpu EPYC' and -cpu Skylake-Server'.
> It's the *host* that makes the difference, not the CPUID presented to
> the guest.
> 
> On the Intel host it boots:
> 
> $ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom 
> ~/Win10_22H2_EnglishInternational_x64v1.iso  -m 16G -M q35 -smp 
> 2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device 
> intel-iommu,intremap=on,dma-translation=off -cpu Skylake-Server --trace 
> vtd_ir_\* --trace apic\* --trace kvm_irqchip\*  
> qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of 
> hotpluggable cpus requested (288) exceeds the recommended cpus supported by 
> KVM (192)
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
> kvm_irqchip_commit_routes 
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.hle [bit 4]
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.rtm [bit 11]
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.hle [bit 4]
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.rtm [bit 11]
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> vtd_ir_enable enable 1
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
> vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
> vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
> vtd_ir_remap_type IOAPIC
> vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 
> 0x40d1)
> kvm_irqchip_update_msi_route Updating MSI route virq=2
> kvm_irqchip_commit_routes 
> vtd_ir_remap_msi_req addr 0xfee00010 data 0xd1 sid 0xffff do_fault 0
> vtd_ir_irte_get index 0 low 0x0 high 0x100d10005
> vtd_ir_remap index 0 trigger 0 vector 209 deliver 0 dest 0x1 mode 1
> vtd_ir_remap_type IOAPIC
> vtd_ir_remap_msi (addr 0xfee00010, data 0xd1) -> (addr 0xfee01004, data 
> 0x40d1)
> kvm_irqchip_update_msi_route Updating MSI route virq=2
> kvm_irqchip_commit_routes 
> ...
> 
> 
> On the AMD host it stops at the boot splash screen:
> 
> $ ./qemu-system-x86_64 -accel kvm,kernel-irqchip=split -cdrom 
> ~/Win10_22H2_EnglishInternational_x64v1.iso  -m 16G -M q35 -smp 
> 2,cores=12,threads=2,maxcpus=288 -accel kvm,kernel-irqchip=split -device 
> intel-iommu,intremap=on,dma-translation=off -cpu  Skylake-Server --trace 
> vtd_ir_\* --trace apic\* --trace kvm_irqchip\* 
> qemu-system-x86_64: -accel kvm,kernel-irqchip=split: warning: Number of 
> hotpluggable cpus requested (288) exceeds the recommended cpus supported by 
> KVM (192)
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
> kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
> kvm_irqchip_commit_routes 
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.hle [bit 4]
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.rtm [bit 11]
> qemu-system-x86_64: warning: This family of AMD CPU doesn't support 
> hyperthreading(2). Please configure -smp options properly or try enabling 
> topoext feature.
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.hle [bit 4]
> qemu-system-x86_64: warning: host doesn't support requested feature: 
> CPUID.07H:EBX.rtm [bit 11]
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> vtd_ir_enable enable 1
> kvm_irqchip_commit_routes 
> kvm_irqchip_commit_routes 
> vtd_ir_remap_msi_req addr 0xfee00004 data 0x0 sid 0xffff do_fault 0
> vtd_ir_remap_msi (addr 0xfee00004, data 0x0) -> (addr 0xfee00004, data 0x0)
> kvm_irqchip_update_msi_route Updating MSI route virq=2
> kvm_irqchip_commit_routes 
> ^Cqemu: terminating on signal 2
> 
> 
> It looks like Windows is putting something bogus into the IOAPIC RTE for irq2?
> In the successful boot that was 0xfee00010 / 0xd1; vector 209 on CPU0.
> 

Hi David,
Do Windows VMs boot with amd-iommu in you setup?  It would be suboptimal in 
performance but is it able to boot?
In my setup, a 360 vcpu Windows VM boot to splash screen and get hung there.

$ qemu-system-x86_64 -vnc :2 --trace vtd_ir_\* --trace apic\* --trace 
kvm_irqchip\* -m 8G -cpu EPYC-Rome -smp 6,cores=60,maxcpus=360 -enable-kvm 
-machine q35,kernel_irqchip=split  -device amd-iommu  w25.qcow2

kvm_irqchip_add_msi_route dev N/A vector 0 virq 0
kvm_irqchip_add_msi_route dev N/A vector 0 virq 1
kvm_irqchip_add_msi_route dev N/A vector 0 virq 2
kvm_irqchip_add_msi_route dev N/A vector 0 virq 3
kvm_irqchip_add_msi_route dev N/A vector 0 virq 4
kvm_irqchip_add_msi_route dev N/A vector 0 virq 5
kvm_irqchip_add_msi_route dev N/A vector 0 virq 6
kvm_irqchip_add_msi_route dev N/A vector 0 virq 7
kvm_irqchip_add_msi_route dev N/A vector 0 virq 8
kvm_irqchip_add_msi_route dev N/A vector 0 virq 9
kvm_irqchip_add_msi_route dev N/A vector 0 virq 10
kvm_irqchip_add_msi_route dev N/A vector 0 virq 11
kvm_irqchip_add_msi_route dev N/A vector 0 virq 12
kvm_irqchip_add_msi_route dev N/A vector 0 virq 13
kvm_irqchip_add_msi_route dev N/A vector 0 virq 14
kvm_irqchip_add_msi_route dev N/A vector 0 virq 15
kvm_irqchip_add_msi_route dev N/A vector 0 virq 16
kvm_irqchip_add_msi_route dev N/A vector 0 virq 17
kvm_irqchip_add_msi_route dev N/A vector 0 virq 18
kvm_irqchip_add_msi_route dev N/A vector 0 virq 19
kvm_irqchip_add_msi_route dev N/A vector 0 virq 20
kvm_irqchip_add_msi_route dev N/A vector 0 virq 21
kvm_irqchip_add_msi_route dev N/A vector 0 virq 22
kvm_irqchip_add_msi_route dev N/A vector 0 virq 23
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes
kvm_irqchip_commit_routes

Regards,
Sandesh