Re: CXL emulation on aarch64

[Itaru Kitayama]

> On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
> wrote:
> 
> On Thu, 16 Jan 2025 15:04:53 +0900
> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> 
>> Hi Jonathan,
>> 
>>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
>>> wrote:
>>> 
>>> On Tue, 14 Jan 2025 12:03:03 +0900
>>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>> 
>>>> Hi Jonathan, 
>>>> 
>>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
>>>>> wrote:
>>>>> 
>>>>> On Fri, 10 Jan 2025 09:20:54 +0000
>>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
>>>>> 
>>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:    
>>>>>>> Hi,
>>>>>>> Is anybody working on the CXL emulation on aarch64?      
>>>>>> 
>>>>>> I'm not currently working on the CXL emulation on aarch64.
>>>>>> 
>>>>>> However, IIRC the CXL maintainer's tree should work.
>>>>>> https://gitlab.com/jic23/qemu/    
>>>>> 
>>>>> Pick up latest branch from there. I'm prepping a rebased version
>>>>> with some new stuff but might take a few more days.    
>>>> 
>>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 
>>>> branches give:
>>>> 
>>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not 
>>>> found  
>>> 
>>> cxl is a machine property not a accel one. So needs to be after virt
>>> There are tests in the tree for bios tables. Copy the command line from 
>>> those.
>>> 
>>>> 
>>>> My commands are below:
>>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
>>>>       -M virt,virtualization=on,gic-version=3 \
>>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
>>>>       -accel tcg,cxl=on \
>>>>       -nographic \
>>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
>>>>       -kernel Image-cca \
>>>>       -drive 
>>>> format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
>>>>       -device virtio-blk-pci,drive=hd0 \
>>>>       -append root=/dev/vda \
>>>>       -nodefaults \
>>>>       --serial tcp:localhost:54320 \
>>>>        -serial tcp:localhost:54321 \
>>>>        -append "root=/dev/vda earlycon console=hvc0" \
>>>>        -device virtio-net-pci,netdev=net0 \
>>>>        -netdev user,id=net0 \
>>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
>>>>        -fsdev local,security_model=none,path=../../,id=shr0
>>>> 
>>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.  
>>> 
>>> I'm a little curious why optee is relevant for this but shouldn't matter as 
>>> long
>>> as an appropriate EDK2 is loaded.
>>> 
>> 
>> I picked up your tree’s “master” and “cxl-next” as of today, and only the 
>> latter at least booted.
>> The former gives:
>> 
>> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
>> 
>> Should I stick with the cxl-next? My concern is that the base QEMU version 
>> is a bit old
>> 7.0.50.
> 
> Always use the latest dated branch on that tree.  I release whenever there
> is something new to carry or a major rebase needed.
> 
> cxl-<date> is the right branch to use. Hope that helps.
> 

Okay the cxl-2024-11-27 gives this:

qemu-system-aarch64: CFMWS does not fit under PA limit

Below is my QEMU options I use currently:

/home/itaru/projects/qemu/build/qemu-system-aarch64 \
         -M 
virt,virtualization=on,pflash0=rom,pflash1=efivars,gic-version=3,virtualization=on,cxl=on
 -m 8192 \
         -cpu cortex-a53 \
         -smp 2 \
         -accel tcg \
         -nographic \
         -display none \
         -kernel ${HOME}/projects/linux/arch/arm64/boot/Image \
         -append "root=/dev/vda rw earlycon acpi=force" \
         -drive format=raw,if=none,file=${HOME}/ubuntu24.img,id=hd0 \
         -device virtio-blk-pci,drive=hd0 \
         -nodefaults \
         -serial mon:stdio \
         -device virtio-net-pci,netdev=net0 \
         -netdev user,id=net0,hostfwd=tcp::8024-:22 \
         -blockdev 
node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
         -blockdev 
node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
         -object 
memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
         -object 
memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
         -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
         -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
         -device 
cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
         -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G

> Jonathan
> 
>> 
>> Thanks,
>> Itaru.
>> 
>>> Jonathan
>>> 
>>>> 
>>>> Let me know which branch you were suggesting.
>>>> 
>>>> Thanks,
>>>> Itaru. 
>>>> 
>>>>> 
>>>>> Note my main development work is on arm64 so that tends to work
>>>>> more reliably than x86 which I only lightly test for stuff that
>>>>> isn't ready for upstream yet.
>>>>> 
>>>>> Give me a shout if you run into any problems.
>>>>> 
>>>>> The main blocker on upstreaming this is resolving the missing device tree
>>>>> support for PCI expander bridges.  I've not made any progress on this 
>>>>> since
>>>>> talk at Linaro connect in 2023.
>>>>> 
>>>>> Jonathan
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Thanks
>>>>>> Zhijian
>>>>>> 
>>>>>>> If there’s a WIP branch, a pointer would be appreciated.
>>>>>>> 
>>>>>>> Itaru