Re: virtio-iommu issue with VFIO device downstream to a PCIe-to-PCI bridge: VFIO devices are not assigned any iommu group

[Jason Wang]

On Tue, Jan 10, 2023 at 5:11 AM Eric Auger <eauger@redhat.com> wrote:
>
> Hi,
>
> On 1/9/23 14:24, Eric Auger wrote:
> > Hi,
> >
> > we have a trouble with virtio-iommu and protected assigned devices
> > downstream to a pcie-to-pci bridge. In that use case we observe the
> > assigned devices are not put to any group. This is true on both x86 and
> > aarch64. This use case works with intel-iommu.
> >
> > *** Guest PCI topology is:
> > lspci -tv
> > -[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM
> > Controller
> >            +-01.0  Device 1234:1111
> >            +-02.0-[01-02]----00.0-[02]----01.0  Broadcom Inc. and
> > subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
> >            +-02.1-[03]--
> >            +-02.2-[04]----00.0  Red Hat, Inc. Virtio block device
> >            +-0a.0  Red Hat, Inc. Device 1057
> >            +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
> >            +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port
> > SATA Controller [AHCI mode]
> >            \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
> >
> >
> > All the assigned devices are aliased and they get devfn=0x0.
> > see qemu pci_device_iommu_address_space in hw/pci.c
> >
> > Initially I see the following traces
> > pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> > pci_device_iommu_address_space name=vfio-pci BDF=0x8 bus=0 devfn=0x8
> > call iommu_fn with bus=0x55f556dde180 and devfn=0
> > virtio_iommu_init_iommu_mr init virtio-iommu-memory-region-0-0
> >
> > Note the bus is 0 at this time and devfn that is used in the
> > virtio-iommu is 0. So an associated IOMMU MR is created with this bus at
> > devfn=0 slot. This is before bus actual numbering.
> >
> > However later on, I see virtio_iommu_probe() and virtio_iommu_attach()
> > getting called with ep_id=520
> > because in the qemu virtio-iommu device, virtio_iommu_mr(pe_id) fails to
> > find the iommu_mr and returns -ENOENT
> >
> > On guest side I see that
> > acpi_iommu_configure_id/iommu_probe_device() fails
> > (__iommu_probe_device) and also __iommu_attach_device would also fail
> > anyway.
> >
> > I guess those get called before actual bus number recomputation?
> >
> > on aarch64 I eventually see the "good" MR beeing created, ie. featuring
> > the right bus number:
> > qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> > BDF=0x208 bus=2 devfn=0x8
> > qemu-system-aarch64: pci_device_iommu_address_space name=vfio-pci
> > BDF=0x208 bus=2 devfn=0x8 call iommu_fn with bus=0xaaaaef12c450 and devfn=0
> >
> > But this does not happen on x86.
> >
> > Jean, do you have any idea about how to fix that? Do you think we have a
> > trouble in the acpi/viot setup or virtio-iommu probe sequence. It looks
> > like virtio probe and attach commands are called too early, before the
> > bus is actually correctly numbered.
>
> So after further investigations looks this is not a problem of bus
> number, which is good at the time of the virtio cmd calls but rather a
> problem related to the devfn (0 was used when creating the IOMMU MR)
> whereas the virtio-iommu cmds looks for the non aliased devfn. With that
> fixed, the probe and attach at least succeeds. The device still does not
> work for me but I will continue my investigations and send a tentative fix.

Haven't thought this deeply, just one thing in my mind and in case
that may help:

intel-iommu doesn't use bus no as the key for hashing address spaces
since it could be configured by the guest:

/*
 * Note that we use pointer to PCIBus as the key, so hashing/shifting
 * based on the pointer value is intended. Note that we deal with
 * collisions through vtd_as_equal().
 */
static guint vtd_as_hash(gconstpointer v)
{
    const struct vtd_as_key *key = v;
    guint value = (guint)(uintptr_t)key->bus;

    return (guint)(value << 8 | key->devfn);
}

Thanks

>
> Thanks
>
> Eric
> >
> > Thanks
> >
> > Eric
> >
> >
> >
> >
> >
> >
> >
>