Re: Quick start guide on RME emulation

[Jean-Philippe Brucker]

On Mon, Oct 23, 2023 at 04:35:35PM -0600, Mathieu Poirier wrote:
> > > QEMU system emulation
> > > ---------------------
> > >
> > > Repo: https://gitlab.com/qemu-project/qemu.git
> > >
> > > Run with:
> > >
> > >         qemu-system-aarch64 -M 
> > > virt,virtualization=on,secure=on,gic-version=3
> > >                 -M acpi=off -cpu max,x-rme=on,sme=off -m 3G -smp 8
> > >                 -nographic
> > >                 -bios tf-a/flash.bin
> > >                 -kernel linux/arch/arm64/boot/Image
> > >                 -drive 
> > > format=raw,if=none,file=buildroot/output/images/rootfs.ext4,id=hd0
> > >                 -device virtio-blk-pci,drive=hd0
> > >                 -append root=/dev/vda
> > >
> > > To use separate consoles for FW (port 54320), secure payload (54321), host
> > > (54322) and guest (54323):
> > >
> > >                 -nodefaults
> > >                 -serial tcp:localhost:54320
> > >                 -serial tcp:localhost:54321
> > >                 -chardev socket,mux=on,id=hvc0,port=54322,host=localhost
> > >                 -device virtio-serial-device
> > >                 -device virtconsole,chardev=hvc0
> > >                 -chardev socket,mux=on,id=hvc1,port=54323,host=localhost
> > >                 -device virtio-serial-device
> > >                 -device virtconsole,chardev=hvc1
> > >                 -append "root=/dev/vda console=hvc0"
> > >                 -device virtio-net-pci,netdev=net0 -net user,id=net0
> >
> > This argument above gave me an error.  See [1], line 19 to see how I
> > corrected it.  Notice the "-netdev user,id=net0" rather than "-net
> > user,id=net0".  Please confirm you have the same on your side.

Yes, I do have '-netdev'

> >
> >
> > Also, "root=/dev/vda" is a duplicate entry.

Right, I left this in case the user wants to keep the default consoles
(the second '-append' overrides the first one). I'll try to clarify that
or get rid of it

> >
> > [1]. https://pastebin.com/KCh8KFD0
> >
> > >
> > >   And connect with 'socat -,rawer TCP-LISTEN:5432x'
> > >
> >
> > There seems to be a syntax error with the above command:
> >
> > $ socat -,rawer TCP-LISTEN:54320
> > 2023/10/19 16:44:55 socat[2799916] E tcsetattr(1, TCSADRAIN,
> > 0x561add6715c0): Invalid argument

I just encountered this one on Ubuntu, it seems to be a bug in socat that
has been fixed in newer versions:
https://bugs.launchpad.net/ubuntu/+source/socat/+bug/1883957

> >
> > I ended up writing my own startup script [2] using the same mechanic
> > as found in OP-TEE.  The "soc_term.spy" file is available here[3].
> >
> > Once the QEMU syntax and the console output were fixed I was able to
> > start the boot stack with the resulting output [4].  The assert, in
> > function gicv2_driver_init()[5] , is caused by variable gic_version
> > being set to 0 rather than 1 or 2.  I found that weird because my QEMU
> > command line [1] explicitly set gic-version=3.  As such I would have
> > expected to read 3.  And why using gic-version=3 when the code is
> > specifically looking for a value of 1 or 2?  Any insight would be
> > appreciated.
> >
> 
> I looked into this problem further today and found my way through.
> Turns out the TF-A defaults to compiling GICv2, and it has to
> explicitly be told to use GICv3, which I did by adding
> QEMU_USE_GIC_DRIVER=QEMU_GICV3 when compiling.  As such, the command
> you have above becomes:
> 
>         make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu
>                  ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40
> QEMU_USE_GIC_DRIVER=QEMU_GICV3
>                  RMM=rmm/build-qemu/Debug/rmm.img
> 
> BL33=edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
>                  all fip
> 
> You problably had to do something like that (or hack the Makefile) but
> simply forgot about it.

Yes, I do have QEMU_USE_GIC_DRIVER=QEMU_GICV3

Thank you for helping with this, I'll fix this doc and move it to the wiki

Thanks,
Jean

>
> With that the TF-A is able to complete its
> initialization and start the Linux kernel in NS world.  Your
> instructions on how to mount the host's rootFS to gain access to the
> realm kernel and CPIO work.  From there I was able to start a realm VM
> and boot it to a command line.  I still haven't seen:
> 
> [    0.000000] RME: RSI version 13.0 advertised
> 
> but that is because I haven't enabled early console access in my
> kernel, something I will do later today.  I also want to start a realm
> VM using the kvmtools.  From there I want to integrate everything in
> the OP-TEE build environment and send you a link for you to try.
> 
> I will get back to you once I have studied the boot stack and the
> enhancement you did to the various components.  Many thanks for
> putting this together, it was a huge help.
> 
> > Regards,
> > Mathieu
> >
> > [2]. https://pastebin.com/XYjUSgSF
> > [3]. https://github.com/OP-TEE/build/blob/master/soc_term.py
> > [4]. https://pastebin.com/qnSXrE7z
> > [5]. 
> > https://jpbrucker.net/git/tf-a/tree/drivers/arm/gic/v2/gicv2_main.c?h=qemu-rme#n128
> >
> > > To share the files needed for booting the guest:
> > >
> > >                 -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
> > >                 -fsdev local,security_model=none,path=.,id=shr0
> > >
> > >   And 'mount -t 9p shr0 /mnt' in the host.
> > >
> > >
> > > Then in the host, launch a Realm guest:
> > >
> > >         qemu-system-aarch64 -M virt,acpi=off,gic-version=3 -nographic
> > >                 -cpu host -enable-kvm -smp 2 -m 512M -overcommit 
> > > mem-lock=on
> > >                 -kernel /mnt/linux/arch/arm64/boot/Image
> > >                 -initrd /mnt/buildroot/output/images/rootfs.cpio
> > >                 -append "console=hvc0 root=/dev/vda"
> > >                 -M confidential-guest-support=rme0
> > >                 -object 
> > > rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256
> > >                 -nodefaults -chardev stdio,mux=on,id=hvc0,signal=off
> > >                 -device virtio-serial-pci -device virtconsole,chardev=hvc0
> > >                 -device virtio-net-pci,netdev=net0,romfile= -netdev 
> > > user,id=net0
> > >                 </dev/hvc1 >/dev/hvc1 &
> > >         ...
> > >         [    0.000000] RME: RSI version 13.0 advertised
> > >