Re: [Qemu-arm] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

[Auger Eric]

Hi,

On 7/11/19 7:28 PM, Eric Auger wrote:
> Up to now vSMMUv3 has not been integrated with VFIO. VFIO
> integration requires to program the physical IOMMU consistently
> with the guest mappings. However, as opposed to VTD, SMMUv3 has
> no "Caching Mode" which allows easy trapping of guest mappings.
> This means the vSMMUV3 cannot use the same VFIO integration as VTD.
> 
> However SMMUv3 has 2 translation stages. This was devised with
> virtualization use case in mind where stage 1 is "owned" by the
> guest whereas the host uses stage 2 for VM isolation.
> 
> This series sets up this nested translation stage. It only works
> if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
> other words, it does not work if there is a physical SMMUv2).
> 
> The series uses a new kernel user API [1] not yet upstream.
> 
> - We force the host to use stage 2 instead of stage 1, when we
>   detect a vSMMUV3 is behind a VFIO device. For a VFIO device
>   without any virtual IOMMU, we still use stage 1 as many existing
>   SMMUs expect this behavior.
> - We use PCIPASIDOps to propage guest stage1 config changes on
>   STE (Stream Table Entry) changes.
> - We implement a specific UNMAP notifier that conveys guest
>   IOTLB invalidations to the host
> - We register MSI IOVA/GPA bindings to the host so that this latter
>   can build a nested stage translation
> - As the legacy MAP notifier is not called anymore, we must make
>   sure stage 2 mappings are set. This is achieved through another
>   prereg memory listener.
> - Physical SMMU stage 1 related faults are reported to the guest
>   via en eventfd mechanism and exposed trhough a dedicated VFIO-PCI
>   region. Then they are reinjected into the guest.
> 
> Best Regards
> 
> Eric
> 
> This series can be found at:
> https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5
Please forgive me. This is the v5 of the RFC. Resending it with the
correct prefix ...

Sorry for the spam :-(

Eric
> 
> Kernel Dependencies:
> [1] [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
> [2] [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part)
> branch at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9
> 
> Includes fixes sent separately:
> [3] [PATCH-for-4.2 v3 0/5] ARM SMMUv3: Fix spurious notification
>     errors and assert with vfio: in case of any comment on those
>     patches, please comment the other series.
> 
> History:
> 
> v4 -> v5:
> - Use PCIPASIDOps for config update notifications
> - removal of notification for MSI binding which is not needed
>   anymore
> - Uses fixes in [3]
> - Use a single fault region
> - use the specific interrupt index
> 
> v3 -> v4:
> - adapt to changes in uapi (asid cache invalidation)
> - check VFIO_PCI_DMA_FAULT_IRQ_INDEX is supported at kernel level
>   before attempting to set signaling for it.
> - sync on 5.2-rc1 kernel headers + Drew's patch that imports sve_context.h
> - fix MSI binding for MSI (not MSIX)
> - fix mingw compilation
> 
> v2 -> v3:
> - rework fault handling
> - MSI binding registration done in vfio-pci. MSI binding tear down called
>   on container cleanup path
> - leaf parameter propagated
> 
> v1 -> v2:
> - Fixed dual assignment (asid now correctly propagated on TLB invalidations)
> - Integrated fault reporting
> 
> Andrew Jones (1):
>   update-linux-headers: Add sve_context.h to asm-arm64
> 
> Eric Auger (27):
>   memory: Remove unused memory_region_iommu_replay_all()
>   memory: Add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute
>   hw/vfio/common: Assert in case of nested mode
>   hw/arm/smmuv3: Log a guest error when decoding an invalid STE
>   hw/arm/smmuv3: Remove spurious error messages on IOVA invalidations
>   update-linux-headers: Import iommu.h
>   header update against 5.3.0-rc0 and IOMMU/VFIO nested stage APIs
>   memory: Add IOMMU_ATTR_MSI_TRANSLATE IOMMU memory region attribute
>   memory: Introduce IOMMU Memory Region inject_faults API
>   memory: Add arch_id and leaf fields in IOTLBEntry
>   iommu: Introduce generic header
>   vfio: Force nested if iommu requires it
>   vfio: Introduce hostwin_from_range helper
>   vfio: Introduce helpers to DMA map/unmap a RAM section
>   vfio: Set up nested stage mappings
>   vfio: Pass stage 1 MSI bindings to the host
>   vfio: Helper to get IRQ info including capabilities
>   vfio/pci: Register handler for iommu fault
>   vfio/pci: Set up the DMA FAULT region
>   vfio/pci: Implement the DMA fault handler
>   hw/arm/smmuv3: Advertise MSI_TRANSLATE attribute
>   hw/arm/smmuv3: Store the PASID table GPA in the translation config
>   hw/arm/smmuv3: Fill the IOTLBEntry arch_id on NH_VA invalidation
>   hw/arm/smmuv3: Fill the IOTLBEntry leaf field on NH_VA invalidation
>   hw/arm/smmuv3: Pass stage 1 configurations to the host
>   hw/arm/smmuv3: Implement fault injection
>   vfio: Remove VFIO/SMMUv3 assert
> 
> Liu Yi L (1):
>   pci: introduce PCIPASIDOps to PCIDevice
> 
>  hw/arm/smmuv3-internal.h                     |   1 +
>  hw/arm/smmuv3.c                              | 195 +++++--
>  hw/arm/trace-events                          |   3 +-
>  hw/pci/pci.c                                 |  34 ++
>  hw/vfio/common.c                             | 517 +++++++++++++++----
>  hw/vfio/pci.c                                | 267 +++++++++-
>  hw/vfio/pci.h                                |   9 +
>  hw/vfio/trace-events                         |   9 +-
>  include/exec/memory.h                        |  59 ++-
>  include/hw/arm/smmu-common.h                 |   1 +
>  include/hw/iommu/iommu.h                     |  28 +
>  include/hw/pci/pci.h                         |  11 +
>  include/hw/vfio/vfio-common.h                |  16 +
>  include/standard-headers/asm-x86/bootparam.h |   2 +
>  include/standard-headers/linux/virtio_ids.h  |   1 -
>  include/standard-headers/linux/virtio_pmem.h |  34 --
>  linux-headers/asm-arm/unistd-common.h        |   1 +
>  linux-headers/asm-arm64/kvm.h                |   7 +
>  linux-headers/asm-generic/unistd.h           |   4 +-
>  linux-headers/asm-mips/unistd_n32.h          |   1 +
>  linux-headers/asm-mips/unistd_n64.h          |   1 +
>  linux-headers/asm-mips/unistd_o32.h          |   1 +
>  linux-headers/asm-powerpc/unistd_32.h        |   1 +
>  linux-headers/asm-powerpc/unistd_64.h        |   1 +
>  linux-headers/asm-s390/unistd_32.h           |   1 +
>  linux-headers/asm-s390/unistd_64.h           |   1 +
>  linux-headers/asm-x86/kvm.h                  |   6 +-
>  linux-headers/asm-x86/unistd_32.h            |   1 +
>  linux-headers/asm-x86/unistd_64.h            |   1 +
>  linux-headers/asm-x86/unistd_x32.h           |   1 +
>  linux-headers/linux/iommu.h                  | 316 ++++++++++++
>  linux-headers/linux/psp-sev.h                |   5 +-
>  linux-headers/linux/vfio.h                   | 109 +++-
>  memory.c                                     |  19 +-
>  scripts/update-linux-headers.sh              |   5 +-
>  35 files changed, 1462 insertions(+), 207 deletions(-)
>  create mode 100644 include/hw/iommu/iommu.h
>  delete mode 100644 include/standard-headers/linux/virtio_pmem.h
>  create mode 100644 linux-headers/linux/iommu.h
>