Re: Quick start guide on RME emulation

[Mathieu Poirier]

Hi Jean-Philippe,

More progress and more debugging...  See below.

On Wed, 4 Oct 2023 at 04:55, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Oct 02, 2023 at 07:52:28PM +0900, Itaru Kitayama wrote:
> > Jean,
> > It is indeed the latter. Itd be good if we could test how far we can go in
> > terms of launching a Realm guest via the CCA aware KVM.
>
> Here is a draft doc that explains how to build and run the whole stack.
> I'm not sure where such a document should go. Maybe the QEMU wiki would be
> best, since these instructions will change frequently.
>
> Note:
>
> * The following instructions are for QEMU virt machine (just because it's
>   the one I'm familiar with). The sbsa-ref machine will be supported.
>
> * I haven't tried to run a secure payload yet, there might be some
>   outstanding bugs. Secure payload such as optee would be packaged with
>   TF-A as BL32.
>
> * I need to rebase the guest edk2 support, so didn't include instructions
>   for that.
>
> Software components:
>
>            |              |              |
>            |    REALM     |  NON-SECURE  |
>            |              |              |
>     -------+--------------+--------------+
>            |              |              |
>       EL0  | Guest Rootfs |  Host Rootfs |
>            |              |  VMM         |
>            |              |              |
>     -------+--------------+--------------+
>            |              |              |
>       EL1  |        EDK2  |              |
>            | Linux Guest  |              |
>            |              |  EDK2        |
>     -------+--------------+  Linux Host  |
>            |              |    (KVM)     |
>       EL2  |      TF-RMM  |              |
>            |              |              |
>     -------+--------------+--------------+
>            |                             |
>       EL3  |            TF-A             |
>            |                             |
>     -------+-----------------------------+
>            |                             |
>       HW   |            QEMU             |
>            |                             |
>     -------+-----------------------------+
>
>
> TF-RMM
> ------
>
> The Realm Management Monitor (RMM) connects KVM and the Realm guest.
>
> RMM gets loaded into NS DRAM (because there isn't enough space in Secure
> RAM). TF-A carves out 24MB of memory for the RMM, and tells other software
> about it using a device-tree reserved memory node.
>
> Status: QEMU support is currently under review. Additional patches are
>   needed until QEMU supports a couple additional features that are
>   mandatory for RME (PMUv3p7 and ECV).
>
> Repo: currently at https://jpbrucker.net/git/rmm/ branch qemu-rme
>       official is https://git.trustedfirmware.org/TF-RMM/tf-rmm.git/
>
> Build:  git submodule update --init --recursive
>         export CROSS_COMPILE=aarch64-linux-gnu-
>         cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg 
> -DRMM_TOOLCHAIN=llvm -B build-qemu
>         cmake --build build-qemu
>
>
> Host EDK2
> ---------
>
> Works out of the box, although it doesn't properly account for reserved
> RMM memory yet.
>
> Repo: https://github.com/tianocore/edk2.git
>
> Build:  git submodule update --init --recursive
>         source edksetup.sh
>         make -j -C BaseTools
>         export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
>         build -b RELEASE -a AARCH64 -t GCC5 -p 
> ArmVirtPkg/ArmVirtQemuKernel.dsc
>
>
> TF-A
> ----
>
> TF-A loads the RMM as well as the Non-secure firmware, and bridges RMM and
> KVM. It also owns the Granule Protection Table (GPT).
>
> Status: QEMU support is currently under review.
>
> Repo: currently at https://jpbrucker.net/git/tf-a/ branch qemu-rme
>       official is https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/
>
> Build:  # Embed the RMM image and edk2 into the Final Image Package (FIP)
>         make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu
>                 ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40
>                 RMM=rmm/build-qemu/Debug/rmm.img
>                 
> BL33=edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
>                 all fip
>
>         # Pack whole image into flash.bin
>         dd if=tfa/build/qemu/debug/bl1.bin of=flash.bin
>         dd if=tfa/build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
>
>
> Host & Guest Linux
> ------------------
>
> Both host and guest need extra patches
>
> Status: 
> https://lore.kernel.org/linux-arm-kernel/20231002124311.204614-1-suzuki.poulose@arm.com/
>
> Repo: https://gitlab.arm.com/linux-arm/linux-cca cca-full/rmm-v1.0-eac2
>
> Build: arm64 defconfig
>
>
> VMM
> ---
>
> Kvmtool and QEMU can be used to launch Realm guests. For kvmtool, see the
> cover letter for the Linux support above.
>
> Status: In development
>
> Repo: for now https://jpbrucker.net/git/qemu/ branch cca/rmm-v1.0-eac2
>
> Build:  git submodule update --init --recursive
>         # to download the missing subprojects because buildroot won't do it:
>         ./configure --target-list=aarch64-softmmu
>         # Note: when buildroot is configured to use this tree (see below),
>         # it will rsync the whole tree including binary files. Therefore,
>         # use a separate build directory to make the system emulation
>         # QEMU, to avoid conflicts between native binary files and
>         # buildroot's cross-build.
>
>
> Rootfs
> ------
>
> Buildroot provides a convenient way to build lightweight root filesystems.
> It can also embed the VMM into the rootfs if you specify the path to
> kvmtool or QEMU source in a local.mk file in the build directory.
>
> Repo: https://gitlab.com/buildroot.org/buildroot.git
>
> local.mk
>         KVMTOOL_OVERRIDE_SRCDIR = path/to/kvmtool/
>         QEMU_OVERRIDE_SRCDIR = path/to/qemu/
>
> Build:  make qemu_aarch64_virt_defconfig
>         # then change some options
>           # BR2_LINUX_KERNEL is not set
>           BR2_PACKAGE_KVMTOOL=y
>           BR2_PACKAGE_QEMU=y
>           BR2_PACKAGE_QEMU_SYSTEM=y
>           # BR2_PACKAGE_QEMU_BLOBS is not set
>           BR2_PACKAGE_QEMU_SLIRP=y
>           BR2_PACKAGE_QEMU_CHOOSE_TARGETS=y
>           BR2_PACKAGE_QEMU_TARGET_AARCH64=y
>           BR2_TARGET_ROOTFS_EXT2_SIZE=256M
>
>           # Generate an initrd for the guest
>           BR2_TARGET_ROOTFS_CPIO=y
>
>         make
>
>
> QEMU system emulation
> ---------------------
>
> Repo: https://gitlab.com/qemu-project/qemu.git
>
> Run with:
>
>         qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3
>                 -M acpi=off -cpu max,x-rme=on,sme=off -m 3G -smp 8
>                 -nographic
>                 -bios tf-a/flash.bin
>                 -kernel linux/arch/arm64/boot/Image
>                 -drive 
> format=raw,if=none,file=buildroot/output/images/rootfs.ext4,id=hd0
>                 -device virtio-blk-pci,drive=hd0
>                 -append root=/dev/vda
>
> To use separate consoles for FW (port 54320), secure payload (54321), host
> (54322) and guest (54323):
>
>                 -nodefaults
>                 -serial tcp:localhost:54320
>                 -serial tcp:localhost:54321
>                 -chardev socket,mux=on,id=hvc0,port=54322,host=localhost
>                 -device virtio-serial-device
>                 -device virtconsole,chardev=hvc0
>                 -chardev socket,mux=on,id=hvc1,port=54323,host=localhost
>                 -device virtio-serial-device
>                 -device virtconsole,chardev=hvc1
>                 -append "root=/dev/vda console=hvc0"
>                 -device virtio-net-pci,netdev=net0 -net user,id=net0

This argument above gave me an error.  See [1], line 19 to see how I
corrected it.  Notice the "-netdev user,id=net0" rather than "-net
user,id=net0".  Please confirm you have the same on your side.  Also,
"root=/dev/vda" is a duplicate entry.

[1]. https://pastebin.com/KCh8KFD0

>
>   And connect with 'socat -,rawer TCP-LISTEN:5432x'
>

There seems to be a syntax error with the above command:

$ socat -,rawer TCP-LISTEN:54320
2023/10/19 16:44:55 socat[2799916] E tcsetattr(1, TCSADRAIN,
0x561add6715c0): Invalid argument

I ended up writing my own startup script [2] using the same mechanic
as found in OP-TEE.  The "soc_term.spy" file is available here[3].

Once the QEMU syntax and the console output were fixed I was able to
start the boot stack with the resulting output [4].  The assert, in
function gicv2_driver_init()[5] , is caused by variable gic_version
being set to 0 rather than 1 or 2.  I found that weird because my QEMU
command line [1] explicitly set gic-version=3.  As such I would have
expected to read 3.  And why using gic-version=3 when the code is
specifically looking for a value of 1 or 2?  Any insight would be
appreciated.

Regards,
Mathieu

[2]. https://pastebin.com/XYjUSgSF
[3]. https://github.com/OP-TEE/build/blob/master/soc_term.py
[4]. https://pastebin.com/qnSXrE7z
[5]. 
https://jpbrucker.net/git/tf-a/tree/drivers/arm/gic/v2/gicv2_main.c?h=qemu-rme#n128

> To share the files needed for booting the guest:
>
>                 -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
>                 -fsdev local,security_model=none,path=.,id=shr0
>
>   And 'mount -t 9p shr0 /mnt' in the host.
>
>
> Then in the host, launch a Realm guest:
>
>         qemu-system-aarch64 -M virt,acpi=off,gic-version=3 -nographic
>                 -cpu host -enable-kvm -smp 2 -m 512M -overcommit mem-lock=on
>                 -kernel /mnt/linux/arch/arm64/boot/Image
>                 -initrd /mnt/buildroot/output/images/rootfs.cpio
>                 -append "console=hvc0 root=/dev/vda"
>                 -M confidential-guest-support=rme0
>                 -object 
> rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256
>                 -nodefaults -chardev stdio,mux=on,id=hvc0,signal=off
>                 -device virtio-serial-pci -device virtconsole,chardev=hvc0
>                 -device virtio-net-pci,netdev=net0,romfile= -netdev 
> user,id=net0
>                 </dev/hvc1 >/dev/hvc1 &
>         ...
>         [    0.000000] RME: RSI version 13.0 advertised
>