Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 22, 2025, at 23:07, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
> wrote:
> 
> On Fri, 17 Jan 2025 09:43:11 +0000
> Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> 
>> On Fri, 17 Jan 2025 10:13:41 +0900
>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>> 
>>>> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
>>>> wrote:
>>>> 
>>>> On Thu, 16 Jan 2025 15:04:53 +0900
>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>> 
>>>>> Hi Jonathan,
>>>>> 
>>>>>> On Jan 14, 2025, at 19:26, Jonathan Cameron 
>>>>>> <Jonathan.Cameron@huawei.com> wrote:
>>>>>> 
>>>>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>>>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>>>>> 
>>>>>>> Hi Jonathan, 
>>>>>>> 
>>>>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron 
>>>>>>>> <Jonathan.Cameron@huawei.com> wrote:
>>>>>>>> 
>>>>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>>>>> 
>>>>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:        
>>>>>>>>>> Hi,
>>>>>>>>>> Is anybody working on the CXL emulation on aarch64?          
>>>>>>>>> 
>>>>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>>>>> 
>>>>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>>>>> https://gitlab.com/jic23/qemu/        
>>>>>>>> 
>>>>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>>>>> with some new stuff but might take a few more days.        
>>>>>>> 
>>>>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 
>>>>>>> branches give:
>>>>>>> 
>>>>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not 
>>>>>>> found      
>>>>>> 
>>>>>> cxl is a machine property not a accel one. So needs to be after virt
>>>>>> There are tests in the tree for bios tables. Copy the command line from 
>>>>>> those.
>>>>>> 
>>>>>>> 
>>>>>>> My commands are below:
>>>>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>>>>      -M virt,virtualization=on,gic-version=3 \
>>>>>>>      -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>>>>      -accel tcg,cxl=on \
>>>>>>>      -nographic \
>>>>>>>      -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>>>>      -kernel Image-cca \
>>>>>>>      -drive 
>>>>>>> format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>>>>      -device virtio-blk-pci,drive=hd0 \
>>>>>>>      -append root=/dev/vda \
>>>>>>>      -nodefaults \
>>>>>>>      --serial tcp:localhost:54320 \
>>>>>>>       -serial tcp:localhost:54321 \
>>>>>>>       -append "root=/dev/vda earlycon console=hvc0" \
>>>>>>>       -device virtio-net-pci,netdev=net0 \
>>>>>>>       -netdev user,id=net0 \
>>>>>>>       -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>>>>       -fsdev local,security_model=none,path=../../,id=shr0
>>>>>>> 
>>>>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.      
>>>>>> 
>>>>>> I'm a little curious why optee is relevant for this but shouldn't matter 
>>>>>> as long
>>>>>> as an appropriate EDK2 is loaded.
>>>>>> 
>>>>> 
>>>>> I picked up your tree’s “master” and “cxl-next” as of today, and only the 
>>>>> latter at least booted.
>>>>> The former gives:
>>>>> 
>>>>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>>>>> 
>>>>> Should I stick with the cxl-next? My concern is that the base QEMU 
>>>>> version is a bit old
>>>>> 7.0.50.    
>>>> 
>>>> Always use the latest dated branch on that tree.  I release whenever there
>>>> is something new to carry or a major rebase needed.
>>>> 
>>>> cxl-<date> is the right branch to use. Hope that helps.    
>>> 
>>> When do you think you want to get them (aarch64 specific?) merged mainline. 
>>> Any reason you want to carry the patches by yourself?  
>> 
>> Nothing much has changed since I presented on this at Linaro connect in 2023.
>> https://resources.linaro.org/en/resource/hM986DSHfoTrZ98UjpvLg1
>> 
>> The issue is device tree bindings for PCI Expander bridgess and the fact that
>> those need to be generated without the full enumeration that EDK2 is doing
>> prior to ACPI final table builds. In order to move forward with that it
>> needs a bunch of work to prove that we absolutely cannot get patches
>> upstream to support kernel base enumeration and breaking up of the
>> various resources (like EDK2 does).
> 
> I was talking to Peter Maydell earlier and given developments in the last 
> couple
> of years that have by necessity been ACPI only in arm virt he is less
> opposed to ACPI only features being added where device tree is challenging.
> 
> So we may be able to move forwards without device tree support.
> 
> The PXB enumeration question is also relevant for managing multiple
> vIOMMUs to represent multiple physical IOMMUs with the correct isolation
> and do it efficiently which is probably a more pressing usecase than CXL 
> emulation.
> The discussion was mainly about that usecase, but maybe it also unblocks
> upstreaming this support.
> 
> Thanks,
> 
> Jonathan

I finally made some CXL tests ran within the ndctl test framework along with 
the kernel modules (QEMU is on your cxl-2024-11-27 branch) on aarch64. However, 
the recent rebase cxl-2025-01-24 fails to start the system emulation

qemu-system-aarch64: Property 'virt-10.0-machine.cxl' not found

The build did not complain, what kind of tests you run against your periodical 
QEMU rebase?

Itaru. 

> 
>> 
>> Given PXB enumeration in kernel has some issues on ARM anyway (that you can 
>> paper
>> over with _DSM 5 - it self requiring an extra patch that isn't upstreamable 
>> because
>> of IO port issues) there is quite a bit of work needed, mostly not in QEMU.
>> Or convince Peter and others that not all virt support needs DT bindings
>> (note that PXB for PCIE has been supported for years without an DT support,
>> just no one noticed!)
>> 
>> After that we'd need to figure out CXL DT bindings in general and add kernel
>> code support - despite there being no known DT based CXL systems out there, 
>> so
>> that is going to be hard to do.  Various CXL kernel maintainers have 
>> expressed
>> they aren't against such support, but it's hardly going to be review priority
>> (other than for me if someone else does the work!)
>> 
>> For me this isn't particularly high priority. The ARM bit is fairly easy to 
>> rebase.
>> I would like to see it solved, but it is behind various other items on my
>> backlog.
>> 
>> There are SBSA machine patches on list, but it's not a useful platform for
>> CXL kernel code development because of the limited supported configurations
>> (in keeping with the more or less fixed model that SBSA-ref uses).
>> 
>> Jonathan
>> 
>> 
>> 
>>> 
>>> Itaru.
>>> 
>>>> 
>>>> Jonathan
>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Itaru.
>>>>> 
>>>>>> Jonathan
>>>>>> 
>>>>>>> 
>>>>>>> Let me know which branch you were suggesting.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Itaru. 
>>>>>>> 
>>>>>>>> 
>>>>>>>> Note my main development work is on arm64 so that tends to work
>>>>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>>>>> isn't ready for upstream yet.
>>>>>>>> 
>>>>>>>> Give me a shout if you run into any problems.
>>>>>>>> 
>>>>>>>> The main blocker on upstreaming this is resolving the missing device 
>>>>>>>> tree
>>>>>>>> support for PCI expander bridges.  I've not made any progress on this 
>>>>>>>> since
>>>>>>>> talk at Linaro connect in 2023.
>>>>>>>> 
>>>>>>>> Jonathan
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> Zhijian
>>>>>>>>> 
>>>>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>>>>> 
>>>>>>>>>> Itaru