RE: [BUG] cxl can not create region

[Dan Williams]

Bobo WL wrote:
> Hi list
> 
> I want to test cxl functions in arm64, and found some problems I can't
> figure out.
> 
> My test environment:
> 
> 1. build latest bios from https://github.com/tianocore/edk2.git master
> branch(cc2db6ebfb6d9d85ba4c7b35fba1fa37fffc0bc2)
> 2. build latest qemu-system-aarch64 from git://git.qemu.org/qemu.git
> master branch(846dcf0ba4eff824c295f06550b8673ff3f31314). With cxl arm
> support patch: 
> https://patchwork.kernel.org/project/cxl/cover/20220616141950.23374-1-Jonathan.Cameron@huawei.com/
> 3. build Linux kernel from
> https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git preview
> branch(65fc1c3d26b96002a5aa1f4012fae4dc98fd5683)
> 4. build latest ndctl tools from https://github.com/pmem/ndctl
> create_region branch(8558b394e449779e3a4f3ae90fae77ede0bca159)
> 
> And my qemu test commands:
> sudo $QEMU_BIN -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 \
>         -cpu max -smp 8 -nographic -no-reboot \
>         -kernel $KERNEL -bios $BIOS_BIN \
>         -drive if=none,file=$ROOTFS,format=qcow2,id=hd \
>         -device virtio-blk-pci,drive=hd -append 'root=/dev/vda1
> nokaslr dyndbg="module cxl* +p"' \
>         -object memory-backend-ram,size=4G,id=mem0 \
>         -numa node,nodeid=0,cpus=0-7,memdev=mem0 \
>         -net nic -net user,hostfwd=tcp::2222-:22 -enable-kvm \
>         -object
> memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M
> \
>         -object
> memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M
> \
>         -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
>         -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
>         -device cxl-upstream,bus=root_port0,id=us0 \
>         -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
>         -device
> cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0 \
>         -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \
>         -device
> cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1 \
>         -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \
>         -device
> cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2 \
>         -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \
>         -device
> cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3 \
>         -M 
> cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k
> 
> And I have got two problems.
> 1. When I want to create x1 region with command: "cxl create-region -d
> decoder0.0 -w 1 -g 4096 mem0", kernel crashed with null pointer
> reference. Crash log:
> 
> [  534.697324] cxl_region region0: config state: 0
> [  534.697346] cxl_region region0: probe: -6
> [  534.697368] cxl_acpi ACPI0017:00: decoder0.0: created region0
> [  534.699115] cxl region0: mem0:endpoint3 decoder3.0 add:
> mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1
> [  534.699149] cxl region0: 0000:0d:00.0:port2 decoder2.0 add:
> mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1
> [  534.699167] cxl region0: ACPI0016:00:port1 decoder1.0 add:
> mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1
> [  534.699176] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256
> [  534.699182] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0
> for mem0:decoder3.0 @ 0
> [  534.699189] cxl region0: 0000:0d:00.0:port2 iw: 1 ig: 256
> [  534.699193] cxl region0: 0000:0d:00.0:port2 target[0] =
> 0000:0e:00.0 for mem0:decoder3.0 @ 0
> [  534.699405] Unable to handle kernel NULL pointer dereference at
> virtual address 0000000000000000
> [  534.701474] Mem abort info:
> [  534.701994]   ESR = 0x0000000086000004
> [  534.702653]   EC = 0x21: IABT (current EL), IL = 32 bits
> [  534.703616]   SET = 0, FnV = 0
> [  534.704174]   EA = 0, S1PTW = 0
> [  534.704803]   FSC = 0x04: level 0 translation fault
> [  534.705694] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010144a000
> [  534.706875] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
> [  534.709855] Internal error: Oops: 86000004 [#1] PREEMPT SMP
> [  534.710301] Modules linked in:
> [  534.710546] CPU: 7 PID: 331 Comm: cxl Not tainted
> 5.19.0-rc3-00064-g65fc1c3d26b9-dirty #11
> [  534.715393] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [  534.717179] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [  534.719190] pc : 0x0
> [  534.719928] lr : commit_store+0x118/0x2cc
> [  534.721007] sp : ffff80000aec3c30
> [  534.721793] x29: ffff80000aec3c30 x28: ffff0000da62e740 x27: 
> ffff0000c0c06b30
> [  534.723875] x26: 0000000000000000 x25: ffff0000c0a2a400 x24: 
> ffff0000c0a29400
> [  534.725440] x23: 0000000000000003 x22: 0000000000000000 x21: 
> ffff0000c0c06800
> [  534.727312] x20: 0000000000000000 x19: ffff0000c1559800 x18: 
> 0000000000000000
> [  534.729138] x17: 0000000000000000 x16: 0000000000000000 x15: 
> 0000ffffd41fe838
> [  534.731046] x14: 0000000000000000 x13: 0000000000000000 x12: 
> 0000000000000000
> [  534.732402] x11: 0000000000000000 x10: 0000000000000000 x9 : 
> 0000000000000000
> [  534.734432] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 
> ffff0000c0906e80
> [  534.735921] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 
> ffff80000aec3bf0
> [  534.737437] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 
> ffff0000c155a000
> [  534.738878] Call trace:
> [  534.739368]  0x0
> [  534.739713]  dev_attr_store+0x1c/0x30
> [  534.740186]  sysfs_kf_write+0x48/0x58
> [  534.740961]  kernfs_fop_write_iter+0x128/0x184
> [  534.741872]  new_sync_write+0xdc/0x158
> [  534.742706]  vfs_write+0x1ac/0x2a8
> [  534.743440]  ksys_write+0x68/0xf0
> [  534.744328]  __arm64_sys_write+0x1c/0x28
> [  534.745180]  invoke_syscall+0x44/0xf0
> [  534.745989]  el0_svc_common+0x4c/0xfc
> [  534.746661]  do_el0_svc+0x60/0xa8
> [  534.747378]  el0_svc+0x2c/0x78
> [  534.748066]  el0t_64_sync_handler+0xb8/0x12c
> [  534.748919]  el0t_64_sync+0x18c/0x190
> [  534.749629] Code: bad PC value
> [  534.750169] ---[ end trace 0000000000000000 ]---

What was the top kernel commit when you ran this test? What is the line
number of "commit_store+0x118"?

> 2. When I want to create x4 region with command: "cxl create-region -d
> decoder0.0 -w 4 -g 4096 -m mem0 mem1 mem2 mem3". I got below errors:
> 
> cxl region: create_region: region0: failed to set target3 to mem3
> cxl region: cmd_create_region: created 0 regions
> 
> And kernel log as below:
> [   60.536663] cxl_region region0: config state: 0
> [   60.536675] cxl_region region0: probe: -6
> [   60.536696] cxl_acpi ACPI0017:00: decoder0.0: created region0
> [   60.538251] cxl region0: mem0:endpoint3 decoder3.0 add:
> mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1
> [   60.538278] cxl region0: 0000:0d:00.0:port2 decoder2.0 add:
> mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1
> [   60.538295] cxl region0: ACPI0016:00:port1 decoder1.0 add:
> mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1
> [   60.538647] cxl region0: mem1:endpoint4 decoder4.0 add:
> mem1:decoder4.0 @ 1 next: none nr_eps: 1 nr_targets: 1
> [   60.538663] cxl region0: 0000:0d:00.0:port2 decoder2.0 add:
> mem1:decoder4.0 @ 1 next: mem1 nr_eps: 2 nr_targets: 2
> [   60.538675] cxl region0: ACPI0016:00:port1 decoder1.0 add:
> mem1:decoder4.0 @ 1 next: 0000:0d:00.0 nr_eps: 2 nr_targets: 1
> [   60.539311] cxl region0: mem2:endpoint5 decoder5.0 add:
> mem2:decoder5.0 @ 2 next: none nr_eps: 1 nr_targets: 1
> [   60.539332] cxl region0: 0000:0d:00.0:port2 decoder2.0 add:
> mem2:decoder5.0 @ 2 next: mem2 nr_eps: 3 nr_targets: 3
> [   60.539343] cxl region0: ACPI0016:00:port1 decoder1.0 add:
> mem2:decoder5.0 @ 2 next: 0000:0d:00.0 nr_eps: 3 nr_targets: 1
> [   60.539711] cxl region0: mem3:endpoint6 decoder6.0 add:
> mem3:decoder6.0 @ 3 next: none nr_eps: 1 nr_targets: 1
> [   60.539723] cxl region0: 0000:0d:00.0:port2 decoder2.0 add:
> mem3:decoder6.0 @ 3 next: mem3 nr_eps: 4 nr_targets: 4
> [   60.539735] cxl region0: ACPI0016:00:port1 decoder1.0 add:
> mem3:decoder6.0 @ 3 next: 0000:0d:00.0 nr_eps: 4 nr_targets: 1
> [   60.539742] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256
> [   60.539747] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0
> for mem0:decoder3.0 @ 0
> [   60.539754] cxl region0: 0000:0d:00.0:port2 iw: 4 ig: 512
> [   60.539758] cxl region0: 0000:0d:00.0:port2 target[0] =
> 0000:0e:00.0 for mem0:decoder3.0 @ 0
> [   60.539764] cxl region0: ACPI0016:00:port1: cannot host mem1:decoder4.0 at 
> 1
> 
> I have tried to write sysfs node manually, got same errors.
> 
> Hope I can get some helps here.

What is the output of:

    cxl list -MDTu -d decoder0.0

...? It might be the case that mem1 cannot be mapped by decoder0.0, or
at least not in the specified order, or that validation check is broken.