Re: CXL emulation on aarch64

[Jonathan Cameron]

On Fri, 17 Jan 2025 09:24:15 +0900
Itaru Kitayama <itaru.kitayama@linux.dev> wrote:

> > On Jan 16, 2025, at 19:58, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
> > wrote:
> > 
> > On Thu, 16 Jan 2025 15:04:53 +0900
> > Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >   
> >> Hi Jonathan,
> >>   
> >>> On Jan 14, 2025, at 19:26, Jonathan Cameron <Jonathan.Cameron@huawei.com> 
> >>> wrote:
> >>> 
> >>> On Tue, 14 Jan 2025 12:03:03 +0900
> >>> Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >>>   
> >>>> Hi Jonathan, 
> >>>>   
> >>>>> On Jan 10, 2025, at 21:31, Jonathan Cameron 
> >>>>> <Jonathan.Cameron@huawei.com> wrote:
> >>>>> 
> >>>>> On Fri, 10 Jan 2025 09:20:54 +0000
> >>>>> "Zhijian Li (Fujitsu)" via <qemu-devel@nongnu.org> wrote:
> >>>>>   
> >>>>>> On 10/01/2025 13:29, Itaru Kitayama wrote:      
> >>>>>>> Hi,
> >>>>>>> Is anybody working on the CXL emulation on aarch64?        
> >>>>>> 
> >>>>>> I'm not currently working on the CXL emulation on aarch64.
> >>>>>> 
> >>>>>> However, IIRC the CXL maintainer's tree should work.
> >>>>>> https://gitlab.com/jic23/qemu/      
> >>>>> 
> >>>>> Pick up latest branch from there. I'm prepping a rebased version
> >>>>> with some new stuff but might take a few more days.      
> >>>> 
> >>>> Thanks for sharing your work with us.  Your master and cxl-2024-11-27 
> >>>> branches give:
> >>>> 
> >>>> $ qemu-system-aarch64: -accel tcg,cxl=on: Property 'tcg-accel.cxl' not 
> >>>> found    
> >>> 
> >>> cxl is a machine property not a accel one. So needs to be after virt
> >>> There are tests in the tree for bios tables. Copy the command line from 
> >>> those.
> >>>   
> >>>> 
> >>>> My commands are below:
> >>>> $HOME/projects/qemu/build/qemu-system-aarch64 \
> >>>>       -M virt,virtualization=on,gic-version=3 \
> >>>>       -M acpi=off -cpu max,sme=off -m 8G -smp 4 \
> >>>>       -accel tcg,cxl=on \
> >>>>       -nographic \
> >>>>       -bios $HOME/cca-v4/out/bin/flash.bin \
> >>>>       -kernel Image-cca \
> >>>>       -drive 
> >>>> format=raw,if=none,file=$HOME/cca-v4/out-or/images/rootfs.ext2,id=hd0 \
> >>>>       -device virtio-blk-pci,drive=hd0 \
> >>>>       -append root=/dev/vda \
> >>>>       -nodefaults \
> >>>>       --serial tcp:localhost:54320 \
> >>>>        -serial tcp:localhost:54321 \
> >>>>        -append "root=/dev/vda earlycon console=hvc0" \
> >>>>        -device virtio-net-pci,netdev=net0 \
> >>>>        -netdev user,id=net0 \
> >>>>        -device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
> >>>>        -fsdev local,security_model=none,path=../../,id=shr0
> >>>> 
> >>>> Yes, I’m using Linaro’s CCA capable OP-TEE builds above.    
> >>> 
> >>> I'm a little curious why optee is relevant for this but shouldn't matter 
> >>> as long
> >>> as an appropriate EDK2 is loaded.
> >>>   
> >> 
> >> I picked up your tree’s “master” and “cxl-next” as of today, and only the 
> >> latter at least booted.
> >> The former gives:
> >> 
> >> qemu-system-aarch64: Property 'virt-9.2-machine.cxl' not found
> >> 
> >> Should I stick with the cxl-next? My concern is that the base QEMU version 
> >> is a bit old
> >> 7.0.50.  
> > 
> > Always use the latest dated branch on that tree.  I release whenever there
> > is something new to carry or a major rebase needed.
> > 
> > cxl-<date> is the right branch to use. Hope that helps.
> >   
> 
> Okay the cxl-2024-11-27 gives this:
> 
> qemu-system-aarch64: CFMWS does not fit under PA limit
> 
> Below is my QEMU options I use currently:
> 

Try using max as the cpu (or n1 or later).  a53 is ancient
which is probably where the limit comes from.

> /home/itaru/projects/qemu/build/qemu-system-aarch64 \
>          -M 
> virt,virtualization=on,pflash0=rom,pflash1=efivars,gic-version=3,virtualization=on,cxl=on
>  -m 8192 \
>          -cpu cortex-a53 \
>          -smp 2 \
>          -accel tcg \
>          -nographic \
>          -display none \
>          -kernel ${HOME}/projects/linux/arch/arm64/boot/Image \
>          -append "root=/dev/vda rw earlycon acpi=force" \
>          -drive format=raw,if=none,file=${HOME}/ubuntu24.img,id=hd0 \
>          -device virtio-blk-pci,drive=hd0 \
>          -nodefaults \
>          -serial mon:stdio \
>          -device virtio-net-pci,netdev=net0 \
>          -netdev user,id=net0,hostfwd=tcp::8024-:22 \
>          -blockdev 
> node-name=rom,driver=file,filename=edk2-aarch64-code.fd,read-only=true \
>          -blockdev 
> node-name=efivars,driver=file,filename=qemu-arm64-efivars.test \
>          -object 
> memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
>          -object 
> memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
>          -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
>          -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
>          -device 
> cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0 \
>          -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G
> 
> > Jonathan
> >   
> >> 
> >> Thanks,
> >> Itaru.
> >>   
> >>> Jonathan
> >>>   
> >>>> 
> >>>> Let me know which branch you were suggesting.
> >>>> 
> >>>> Thanks,
> >>>> Itaru. 
> >>>>   
> >>>>> 
> >>>>> Note my main development work is on arm64 so that tends to work
> >>>>> more reliably than x86 which I only lightly test for stuff that
> >>>>> isn't ready for upstream yet.
> >>>>> 
> >>>>> Give me a shout if you run into any problems.
> >>>>> 
> >>>>> The main blocker on upstreaming this is resolving the missing device 
> >>>>> tree
> >>>>> support for PCI expander bridges.  I've not made any progress on this 
> >>>>> since
> >>>>> talk at Linaro connect in 2023.
> >>>>> 
> >>>>> Jonathan
> >>>>> 
> >>>>>   
> >>>>>> 
> >>>>>> 
> >>>>>> Thanks
> >>>>>> Zhijian
> >>>>>>   
> >>>>>>> If there’s a WIP branch, a pointer would be appreciated.
> >>>>>>> 
> >>>>>>> Itaru        
> 
>