[Bug 1921444] Re: Q35 doesn't support to hot add the 2nd PCIe device to KVM guest

[Thomas Huth]

The QEMU project is currently moving its bug tracking to another system.
For this we need to know which bugs are still valid and which could be
closed already. Thus we are setting the bug state to "Incomplete" now.

If the bug has already been fixed in the latest upstream version of QEMU,
then please close this ticket as "Fix released".

If it is not fixed yet and you think that this bug report here is still
valid, then you have two options:

1) If you already have an account on gitlab.com, please open a new ticket
for this problem in our new tracker here:

    https://gitlab.com/qemu-project/qemu/-/issues

and then close this ticket here on Launchpad (or let it expire auto-
matically after 60 days). Please mention the URL of this bug ticket on
Launchpad in the new ticket on GitLab.

2) If you don't have an account on gitlab.com and don't intend to get
one, but still would like to keep this ticket opened, then please switch
the state back to "New" or "Confirmed" within the next 60 days (other-
wise it will get closed as "Expired"). We will then eventually migrate
the ticket automatically to the new system (but you won't be the reporter
of the bug in the new system and thus you won't get notified on changes
anymore).

Thank you and sorry for the inconvenience.


** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1921444

Title:
  Q35 doesn't support to hot add the 2nd PCIe device to KVM guest

Status in QEMU:
  Incomplete

Bug description:
  KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git  branch: next, commit: 
4a98623d
  Qemu: https://git.qemu.org/git/qemu.git  branch: master, commit: 9e2e9fe3

  Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device
  to guest with qemu internal command device_add, the 1st device can be
  added successfully, but the 2nd device failed to hot add.

  If guest chipset is legacy i440fx, the 2 device can be added
  successfully.

  1. Enable VT-d in BIOS
  2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel
  3. Bind 2 device to vfio-pci
      echo 0000:b1:00.0 > /sys/bus/pci/drivers/i40e/unbind
      echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 
      echo 0000:b1:00.1 > /sys/bus/pci/drivers/i40e/unbind
      echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id 

  4. create guest with Q35 chipset:
  qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive 
file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device 
virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device 
pcie-root-port,id=root1 -daemonize

  5. hot add the 1st device to guest successfully
  in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1"
  6. hot add the 2nd device to guest
  in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1"
  The 2nd device doesn't be added in guest, and the 1st device is removed from 
guest. 

  Guest partial log:
  [  110.452272] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button 
pressed
  [  110.453314] pcieport 0000:00:04.0: pciehp: Slot(0) Powering on due to 
button press
  [  110.454156] pcieport 0000:00:04.0: pciehp: Slot(0): Card present
  [  110.454792] pcieport 0000:00:04.0: pciehp: Slot(0): Link Up
  [  110.580927] pci 0000:01:00.0: [8086:1572] type 00 class 0x020000
  [  110.582560] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x007fffff 64bit 
pref]
  [  110.583453] pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00007fff 64bit 
pref]
  [  110.584278] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
  [  110.585051] pci 0000:01:00.0: Max Payload Size set to 128 (was 512, max 
2048)
  [  110.586621] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
  [  110.588140] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00800000 
64bit pref]
  [  110.588954] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00800000 
64bit pref]
  [  110.589797] pci 0000:01:00.0: BAR 6: assigned [mem 0xfe800000-0xfe87ffff 
pref]
  [  110.590703] pci 0000:01:00.0: BAR 3: assigned [mem 0xfe000000-0xfe007fff 
64bit pref]
  [  110.592085] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.592755] pcieport 0000:00:04.0:   bridge window [io  0x1000-0x1fff]
  [  110.594403] pcieport 0000:00:04.0:   bridge window [mem 
0xfe800000-0xfe9fffff]
  [  110.595847] pcieport 0000:00:04.0:   bridge window [mem 
0xfe000000-0xfe1fffff 64bit pref]
  [  110.597867] PCI: No. 2 try to assign unassigned res
  [  110.597870] release child resource [mem 0xfe000000-0xfe007fff 64bit pref]
  [  110.597871] pcieport 0000:00:04.0: resource 15 [mem 0xfe000000-0xfe1fffff 
64bit pref] released
  [  110.598881] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.600789] pcieport 0000:00:04.0: BAR 15: assigned [mem 
0x180000000-0x180bfffff 64bit pref]
  [  110.601731] pci 0000:01:00.0: BAR 0: assigned [mem 0x180000000-0x1807fffff 
64bit pref]
  [  110.602849] pci 0000:01:00.0: BAR 3: assigned [mem 0x180800000-0x180807fff 
64bit pref]
  [  110.604069] pcieport 0000:00:04.0: PCI bridge to [bus 01]
  [  110.604941] pcieport 0000:00:04.0:   bridge window [io  0x1000-0x1fff]
  [  110.606237] pcieport 0000:00:04.0:   bridge window [mem 
0xfe800000-0xfe9fffff]
  [  110.607401] pcieport 0000:00:04.0:   bridge window [mem 
0x180000000-0x180bfffff 64bit pref]
  [  110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver
  [  110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
  [  110.655314] i40e 0000:01:00.0: enabling device (0140 -> 0142)
  [  110.672396] i40e 0000:01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 
1.1747.0 [8086:1572] [8086:0008]
  [  110.750054] i40e 0000:01:00.0: MAC address: 3c:fd:fe:c0:59:98
  [  110.751792] i40e 0000:01:00.0: FW LLDP is enabled
  [  110.764644] i40e 0000:01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, 
Flow Control: None
  [  110.779390] i40e 0000:01:00.0: PCI-Express: Speed 8.0GT/s Width x8
  [  110.789841] i40e 0000:01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 
RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
  [  111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
  [  205.130288] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button 
pressed
  [  205.131743] pcieport 0000:00:04.0: pciehp: Slot(0): Powering off due to 
button press
  [  205.133233] pcieport 0000:00:04.0: pciehp: Slot(0): Card not present
  [  205.135728] i40e 0000:01:00.0: i40e_ptp_stop: removed PHC on eth1

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1921444/+subscriptions