Re: [Qemu-arm] [Qemu-devel] [RFC v4 00/27] vSMMUv3/pSMMUv3 2 stage VFIO integration

[Auger Eric]

Hi Zhangfei,

On 7/11/19 3:53 AM, Zhangfei Gao wrote:
> On Mon, May 27, 2019 at 7:44 PM Eric Auger <address@hidden> wrote:
>>
>> Up to now vSMMUv3 has not been integrated with VFIO. VFIO
>> integration requires to program the physical IOMMU consistently
>> with the guest mappings. However, as opposed to VTD, SMMUv3 has
>> no "Caching Mode" which allows easy trapping of guest mappings.
>> This means the vSMMUV3 cannot use the same VFIO integration as VTD.
>>
>> However SMMUv3 has 2 translation stages. This was devised with
>> virtualization use case in mind where stage 1 is "owned" by the
>> guest whereas the host uses stage 2 for VM isolation.
>>
>> This series sets up this nested translation stage. It only works
>> if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
>> other words, it does not work if there is a physical SMMUv2).
>>
>> The series uses a new kernel user API [1], still under definition.
>>
>> - We force the host to use stage 2 instead of stage 1, when we
>>   detect a vSMMUV3 is behind a VFIO device. For a VFIO device
>>   without any virtual IOMMU, we still use stage 1 as many existing
>>   SMMUs expect this behavior.
>> - We introduce new IOTLB "config" notifiers, requested to notify
>>   changes in the config of a given iommu memory region. So now
>>   we have notifiers for IOTLB changes and config changes.
>> - vSMMUv3 calls config notifiers when STE (Stream Table Entries)
>>   are updated by the guest.
>> - We implement a specific UNMAP notifier that conveys guest
>>   IOTLB invalidations to the host
>> - We implement a new MAP notifiers only used for MSI IOVAs so
>>   that the host can build a nested stage translation for MSI IOVAs
>> - As the legacy MAP notifier is not called anymore, we must make
>>   sure stage 2 mappings are set. This is achieved through another
>>   memory listener.
>> - Physical SMMUs faults are reported to the guest via en eventfd
>>   mechanism and reinjected into this latter.
>>
>> Note: The first patch is a code cleanup and was sent separately.
>>
>> Best Regards
>>
>> Eric
>>
>> This series can be found at:
>> https://github.com/eauger/qemu/tree/v4.0.0-2stage-rfcv4
>>
>> Compatible with kernel series:
>> [PATCH v8 00/29] SMMUv3 Nested Stage Setup
>> (https://lkml.org/lkml/2019/5/26/95)
>>
> 
> Have tested vfio mode in qemu on arm64 platform.
> 
> Tested-by: Zhangfei Gao <address@hidden>
> qemu: https://github.com/eauger/qemu/tree/v4.0.0-2stage-rfcv4
> kernel: https://github.com/eauger/linux/tree/v5.2-rc1-2stage-v8

Your testing is really appreciated.

Both kernel and QEMU series will be respinned. I am currently waiting
for 5.3 kernel window as it will resolve some dependencies on the fault
reporting APIs. My focus is to get the updated kernel series reviewed
and tested and then refine the QEMU integration accordingly.

Thanks

Eric


>