Re: [RESEND] Multiple SMMUv3 instances on PCI Bus and PCI Host Bridge

[Eric Auger]

Hi,
On 6/5/21 1:16 AM, Nicolin Chen wrote:
> (Resending as my previous mail didn't go through mailing lists)
>
> Hello Eric, Yubo, and other QEMU developers,
>
> I am having a problem with links between vSMMU and PCI Host Bridge,
> using ARM-VIRT (64-bit; ACPI) + SMMUv3 (nested translation) setup.
>
> Firstly, I am very new to the areas of QEMU, PCI and ACPI. So some
> of my thoughts/ideas below might not sound very reasonable to you.
>
> My goal here is to create two vSMMU instances in QEMU level and to
> link them to different passthrough devices: each vSMMU has my local
> feature that reads/writes through a VFIO mdev interface to talk to
> Host OS, so it has to be two vSMMU instances in the QEMU level for
> my use case.
>
> As we know, QEMU by default has only one PCI root bus (PCIE.0) that
> links to a default vSMMU (let's call it vSMMU0). And I learned that
> now ARM-VIRT has PCI gpex feature. So I was planning to create one
> host bridge (PCIE.128) to link to a different instance (vSMMU1) --
> later on I can pass through different PCI devices to either PCIE.0
> or PCIE.128 for different mdev pathways.
>
> I then tried to add a PCI Host Bridge using the following commands
> that created one default vSMMU instance, as my first experiment.
>
> /home/ubuntu/qemu-system-aarch64
>     -machine virt,accel=kvm,gic-version=3,iommu=smmuv3 \
>     -cpu host -smp cpus=1 -m 1G -nographic -monitor none -display none \
>     -kernel /boot/Image -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>     -initrd /home/ubuntu/buildroot-20200422-aarch64-qemu-test-rootfs.cpio \
>     -object memory-backend-ram,size=1G,id=m0 \
>     -numa node,cpus=0,nodeid=0,memdev=m0 \
>     -device pxb-pcie,id=pxb-pcie.128,bus=pcie.0,bus_nr=128,numa_node=0 \
>     -device 
> pcie-root-port,id=pcie.128,bus=pxb-pcie.128,slot=1,addr=0,io-reserve=0 \
>     -device vfio-pci,host=0003:01:00.0,rombar=0,bus=pcie.128
>
> However I found that PCIE.128 was also added to vSMMU0, which feels
> like that PCIE.128 treated PCIE.0 root bus as a parent device so it
> was added to the parent's vSMMU too.
>
> Then I tried another experiment with the following hack, hoping that
> it would link vSMMU0 to PCIE.128 instead of PCIE.0:
>
> @@ -385,13 +387,13 @@ build_iort(GArray *table_data, BIOSLinker *linker, 
> VirtMachineState *vms)
>      /* fully coherent device */
>      rc->memory_properties.cache_coherency = cpu_to_le32(1);
>      rc->memory_properties.memory_flags = 0x3; /* CCA = CPM = DCAS = 1 */
>      rc->pci_segment_number = bus_num; /* MCFG pci_segment */
> +    rc->pci_segment_number = cpu_to_le32(bus_num); /* MCFG pci_segment */
>
>      /* Identity RID mapping covering the whole input RID range */
>      idmap = &rc->id_mapping_array[0];
>      idmap->input_base = 0;
>      idmap->id_count = cpu_to_le32(0xFFFF);
> -    idmap->output_base = 0;
> +    idmap->output_base = cpu_to_le32(bus_num << 16);
>
> Yet, I found it's not successful either: the vSMMU instance was not
> added to either PCIE.0 or PCIE.128.
>
> So I started to have questions in my mind:
> (1) Can PCI host bridge (PCIE.128) add to a different vSMMU without
>     following PCIE.0's SMMU setup?
changes need to be made in hw/arm/virt.c
create_smmu() is passed the primary bus the iommu is attached to.
Currently arm virt only supports one smmu instance. So playing with qemu
options is not sufficient.
Besides that, effectively there are IORT changes needed because you need
to route RCs to the different SMMU instances, ie. some RIDs need to
reach SMMU#0 and others #SMMU#1.
You can get inspired of "[PATCH v4 6/8] hw/arm/virt-acpi-build: Add
explicit IORT idmap for smmuv3 node for this kind of changes."

Thanks

Eric
> (2) If the answer to (1) is yes, is there any way to have two pairs
>     of PCI+vSMMU?
> (3) If the answer to (1) is no, how can I correctly change the iort
>     table to link vSMMU to PCIE.128?
>
> Thanks
> Nic
>