PCI hot plug to a VM does not work in Alma

[Chanda Mendon (cmendon)]

Hi
We have a chassis with a peripheral PCI device installed. We have a hypervisor running on the chassis where we have deployed a VM which can use the PCI device once it is attached. 

When the PCI device is powered on or off we need to do a hotplug in/out using virsh commands. Even though the virsh commands for hotplug is executed successfully, the VM sees the PCI inside the VM for one or 2 secs. What do you think is the issue?
[root ~]# uname -a

Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Mar 15 05:32:38 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[root ~]# virsh --version

8.0.0

[root ~]# rpm -qa | grep libvirt

libvirt-daemon-driver-nwfilter-8.0.0-5.el8.x86_64

python3-libvirt-7.8.0-1.el8.x86_64

libvirt-daemon-driver-storage-logical-8.0.0-5.el8.x86_64

libvirt-libs-8.0.0-5.el8.x86_64

libvirt-daemon-config-nwfilter-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-gluster-8.0.0-5.el8.x86_64

libvirt-8.0.0-5.el8.x86_64

libvirt-daemon-8.0.0-5.el8.x86_64

libvirt-daemon-driver-nodedev-8.0.0-5.el8.x86_64

libvirt-daemon-config-network-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-iscsi-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-rbd-8.0.0-5.el8.x86_64

libvirt-daemon-driver-network-8.0.0-5.el8.x86_64

libvirt-daemon-driver-secret-8.0.0-5.el8.x86_64

python2-libvirt-python-5.10.0-1.el8.x86_64

libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-core-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-iscsi-direct-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-scsi-8.0.0-5.el8.x86_64

libvirt-client-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-disk-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-8.0.0-5.el8.x86_64

libvirt-daemon-driver-interface-8.0.0-5.el8.x86_64

libvirt-daemon-driver-storage-mpath-8.0.0-5.el8.x86_64

libvirt-daemon-kvm-8.0.0-5.el8.x86_64

[root ~]# rpm -qa | grep qemu

qemu-kvm-ui-opengl-6.2.0-11.el8.x86_64

qemu-kvm-6.2.0-11.el8.x86_64

qemu-img-6.2.0-11.el8.x86_64

qemu-kvm-block-iscsi-6.2.0-11.el8.x86_64

ipxe-roms-qemu-20200823-7.git4bd064de.el8.noarch

qemu-kvm-block-gluster-6.2.0-11.el8.x86_64

qemu-kvm-block-rbd-6.2.0-11.el8.x86_64

qemu-kvm-block-curl-6.2.0-11.el8.x86_64

qemu-kvm-core-6.2.0-11.el8.x86_64

qemu-kvm-hw-usbredir-6.2.0-11.el8.x86_64

libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64

qemu-kvm-ui-spice-6.2.0-11.el8.x86_64

qemu-kvm-docs-6.2.0-11.el8.x86_64

qemu-kvm-block-ssh-6.2.0-11.el8.x86_64

qemu-kvm-common-6.2.0-11.el8.x86_64

 

[root ~]# virsh nodedev-dettach  pci_0000_04_00_0

[12013.987821] pci_probe_reset_slot: call pci_slot_reset with probe=1

[12014.063669] pci_slot_reset (printk info): reset hotplug slot.

[12014.134164] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[Mar19 21:09] pci_probe_reset_slot: call pci_slot_reset with probe=1

[  +0.075848] pci_slot_reset (printk info): reset hotplug slot.

[  +0.070495] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

Device pci_0000_04_00_0 detached

 

[root ~]# virsh attach-device ROUTER8 /opt/us/bin/mrvl.xml

[12024.217540] pci_probe_reset_slot: call pci_slot_reset with probe=1

[ +10.083376] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.293548] pci_slot_reset (printk info): reset hotplug slot.

 

[12024.434828] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.076008] pci_slot_reset (printk info): reset hotplug slot.

[  +0.141280] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)4.526807] pci_probe_reset_slot: call pci_slot_reset with probe=1

 

[12024.749367] pci_slot_reset (printk info): reset hotplug slot.

[12024.821327] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.091979] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.911065] pci_probe_reset_slot: call pci_slot_reset with probe=1

 

[  +0.222560] pci_slot_reset (printk info): reset hotplug slot.

[  +0.071960] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

5.057584] pci_slot_reset (printk info): reset hotplug slot.

[  +0.089738] pci_probe_reset_slot: call pci_slot_reset with probe=1

[12025.278585] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.146519] pci_slot_reset (printk info): reset hotplug slot.

[  +0.221001] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

5.441574] pci_probe_reset_slot: call pci_slot_reset with probe=1

[  +0.162989] pci_probe_reset_slot: call pci_slot_reset with probe=1[12025.665968] pci_slot_reset (printk info): reset hotplug slot.

 

[12025.806751] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[12025.898060] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.

[  +0.224394] pci_slot_reset (printk info): reset hotplug slot.

[  +0.140783] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)5.988477] pci_slot_reset (printk info): reset hotplug slot.

 

[  +0.091309] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.

12026.205654] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[12026.383424] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot.

[  +0.090417] pci_slot_reset (printk info): reset hotplug slot.

[  +0.217177] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)6.490406] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0)

 

[  +0.177770] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot.

2026.726579] pciehp_reset_slot: SLOTCTRL 58 write cmd 0

[  +0.106982] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0)

[  +0.236167] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 0

[  +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 0

[12027.744480] pending interrupts 0x0108 from Slot Status

[  +1.017890] pcieport 0000:00:0e.0: pciehp: pending interrupts 0x0108 from Slot Status

[  +0.000011] pending interrupts 0x0108 from Slot Status

[12027.936469] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008

[  +0.191983] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 100812028.027182] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down

 

[12028.190357] Slot(4): Link Down

[  +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008[12028.230605] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present

[12028.369819] Slot(4): Card not present

 

Device attached successfully

[12028.415302] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00

[  +0.090713] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down[12028.524356] vfio-pci 0000:04:00.0: Relaying device request to user (#0)

 

 

[  +0.163175] Slot(4): Link Down

[  +0.040248] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present

[  +0.139214] Slot(4): Card not present

[  +0.045479] pcieport 0000:00:0e.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:04:00

[  +0.000004] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00

[  +0.109054] vfio-pci 0000:04:00.0: Relaying device request to user (#0)

9119] pci_probe_reset_slot: call pci_slot_reset with probe=1

[root@nfvis ~]# [12029.151422] pci_slot_reset (printk info): reset hotplug slot.

[12029.237806] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.334763] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.327572] pci_probe_reset_slot: call pci_slot_reset with probe=1

 

[  +0.292303] pci_slot_reset (printk info): reset hotplug slot.

[  +0.086384] pci_r12029.474447] pci_slot_reset (printk info): reset hotplug slot.

eset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[12029.632533] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.089766] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.788423] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.

 

[  +0.146875] pci_slot_reset (printk info): reset hotplug slot.

[12029.950782] pci_slot_reset (printk info): reset hotplug slot.

[  +0.158086] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[  +0.155890] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1.

.089787] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)

[12030.346708] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot.

[  +0.162359] pci_slot_reset (printk info): reset hotplug slot.[12030.451555] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible)

 

[  +0.139005] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)12030.632974] pci 0000:04:00.0: Removing from iommu group 58

 

[  +0.256921] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot.

[  +0.104847] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible)

5279] pciehp_check_link_active: lnk_status = 7011

[  +0.181419] pci 0000:04:00.0: Removing from iommu group 58

[12031.056747] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present

[12031.194871] Slot(4): Card present

[  +0.152301] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_active: lnk_status = 701112031.236173] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up

[12031.395181] Slot(4): Link Up

 

[  +0.000004] pciehp_check_link_active: lnk_status = 7011

[  +0.271468] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present

[  +0.138124] Slot(4): Card present

[  +0.041302] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up

[  +0.159008] Slot(4): Link Up

31.560448] read_dev_vendor_id: read config_dword worked.  bus = 4, dev_vendor id = 0xE61E11AB

[12031.803909] read_dev_vendor_id: device found, return true.  bus = 4, dev_vendor id = 0xE61E11AB

[12031.909784] pciehp_check_link_status: lnk_status = 7011

[  +0.165267] read_dev_vendor_id: read config_dword worked.  bus = 4, dev_vendor id = 0xE61E11AB2031.974018] read_dev_vendor_id: read config_dword worked.  bus = 4, dev_vendor id = 0xE61E11AB

[12032.176762] read_dev_vendor_id: device found, return true.  bus = 4, dev_vendor id = 0xE61E11AB

 

[  +0.243461] read_dev_vendor_id: device found, return true.  bus = 4, dev_vendor id = 0xE61E11AB

[  +0.105871] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_status: lnk_status = 7011282681] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000

 

[  +0.000004] pciehp_check_link_status: lnk_status = 7011

[12032.545840] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref]

[12032.701716] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref]

[  +0.064234] read_dev_vendor_id: read config_dword worked.  bus = 4, dev_vendor id = 0xE61E11AB2032.794062] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref]

 

[12032.984416] pci 0000:04:00.0: supports D1 D2

[  +0.202744] read_dev_vendor_id: device found, return true.  bus = 4, dev_vendor id = 0xE61E11AB2033.039350] pci 0000:04:00.0: Adding to iommu group 58

[12033.201376] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref]

 

[12033.299989] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref]

[  +0.105919] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000[12033.400664] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref]

 

[  +0.263159] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref]12033.568019] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06]

[12033.725967] pcieport 0000:00:0e.0:   bridge window [io  0xd000-0xdfff]

 

[  +0.155876] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref]12033.805813] pcieport 0000:00:0e.0:   bridge window [mem 0xdc000000-0xdcffffff]

[12033.982533] pcieport 0000:00:0e.0:   bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref]

 

[  +0.092346] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref]12034.086495] pci-stub 0000:04:00.0: claimed by stub

 

[  +0.190354] pci 0000:04:00.0: supports D1 D2

[  +0.054934] pci 0000:04:00.0: Adding to iommu group 58

[  +0.162026] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref]

[  +0.098613] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref]

[  +0.100675] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref]

[  +0.167355] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06]

[  +0.157948] pcieport 0000:00:0e.0:   bridge window [io  0xd000-0xdfff]

[  +0.079846] pcieport 0000:00:0e.0:   bridge window [mem 0xdc000000-0xdcffffff]

[  +0.176720] pcieport 0000:00:0e.0:   bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref]

[  +0.103962] pci-stub 0000:04:00.0: claimed by stub

 

Thank you for replying in advance,

-Chanda