pci passhthrough device's pci mem cannot mapped

[Jiatong Shen]

Hello community experts,

  I am using a 4.2.0 versioned qemu and passthrough an A40 (48G GPU memory). When I start the virtual machine with following option "-m size=4096M,slots=2,maxmem=1T ", the virtual machine could boot but the in the VM I saw lspci failed to find bar1 and kernel message attached in the end of the mail.

root@a40-test:~# lspci -s 00:08.0 -vvv

00:08.0 3D controller: NVIDIA Corporation Device 2235 (rev a1)

Subsystem: NVIDIA Corporation Device 145a

Physical Slot: 8

Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Interrupt: pin A routed to IRQ 11

Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]

Region 1: Memory at <ignored> (64-bit, prefetchable)

Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]

I tried 6.2.0 with the same options and it works without problem. So my question what might cause the problem and why. Thank you very much for help.

--

Best Regards,

Jiatong Shen

kernel message:

[mem 0xc0000000-0xfeffbfff] available for PCI devices

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: [10de:2235] type 00 class 0x030200

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: reg 0x10: [mem 0xfb000000-0xfbffffff]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: reg 0x14: [mem 0x11000000000-0x11fffffffff 64bit pref]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: reg 0x1c: [mem 0x12000000000-0x12001ffffff 64bit pref]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: Enabling HDA controller

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: PME# supported from D0 D3hot

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x16 link at 0000:00:03.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: can't claim BAR 1 [mem 0x11000000000-0x11fffffffff 64bit pref]: no compatible bridge window

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: can't claim BAR 3 [mem 0x12000000000-0x12001ffffff 64bit pref]: no compatible bridge window

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: BAR 1: no space for [mem size 0x1000000000 64bit pref]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: BAR 1: trying firmware assignment [mem 0x11000000000-0x11fffffffff 64bit pref]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: BAR 1: [mem 0x11000000000-0x11fffffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0xffffffffff]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: BAR 1: failed to assign [mem size 0x1000000000 64bit pref]

[Thu Oct 26 15:16:42 2023] pci 0000:00:03.0: BAR 3: assigned [mem 0xc0000000-0xc1ffffff 64bit pref]

NVRM: BAR1 is 0M @ 0x0 (PCI:0000:00:03.0)

[Thu Oct 26 15:16:46 2023] nvidia: probe of 0000:00:03.0 failed with error -1

NVRM: BAR1 is 0M @ 0x0 (PCI:0000:00:03.0)

[Thu Oct 26 15:16:47 2023] nvidia: probe of 0000:00:03.0 failed with error -1

My command looks like ,

qemu-system-x86_64 -cpu host -smp 2 -m 4096 \

-enable-kvm -device vfio-pci,host=0000:4b:00.0 \

-drive file=test,if=virtio,format=raw \

-netdev user,id=net0,hostfwd=tcp::122222-:22 \

-device virtio-net-pci,netdev=net0 \

-vnc 0.0.0.0:70