Re: Quick start guide on RME emulation

[Mathieu Poirier]

Hi Jean-philippe,

Following the steps herein I have the following questions - please see below.

On Wed, 4 Oct 2023 at 04:55, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Oct 02, 2023 at 07:52:28PM +0900, Itaru Kitayama wrote:
> > Jean,
> > It is indeed the latter. Itd be good if we could test how far we can go in
> > terms of launching a Realm guest via the CCA aware KVM.
>
> Here is a draft doc that explains how to build and run the whole stack.
> I'm not sure where such a document should go. Maybe the QEMU wiki would be
> best, since these instructions will change frequently.
>
> Note:
>
> * The following instructions are for QEMU virt machine (just because it's
>   the one I'm familiar with). The sbsa-ref machine will be supported.
>
> * I haven't tried to run a secure payload yet, there might be some
>   outstanding bugs. Secure payload such as optee would be packaged with
>   TF-A as BL32.
>
> * I need to rebase the guest edk2 support, so didn't include instructions
>   for that.
>
> Software components:
>
>            |              |              |
>            |    REALM     |  NON-SECURE  |
>            |              |              |
>     -------+--------------+--------------+
>            |              |              |
>       EL0  | Guest Rootfs |  Host Rootfs |
>            |              |  VMM         |
>            |              |              |
>     -------+--------------+--------------+
>            |              |              |
>       EL1  |        EDK2  |              |
>            | Linux Guest  |              |
>            |              |  EDK2        |
>     -------+--------------+  Linux Host  |
>            |              |    (KVM)     |
>       EL2  |      TF-RMM  |              |
>            |              |              |
>     -------+--------------+--------------+
>            |                             |
>       EL3  |            TF-A             |
>            |                             |
>     -------+-----------------------------+
>            |                             |
>       HW   |            QEMU             |
>            |                             |
>     -------+-----------------------------+
>
>
> TF-RMM
> ------
>
> The Realm Management Monitor (RMM) connects KVM and the Realm guest.
>
> RMM gets loaded into NS DRAM (because there isn't enough space in Secure
> RAM). TF-A carves out 24MB of memory for the RMM, and tells other software
> about it using a device-tree reserved memory node.
>
> Status: QEMU support is currently under review. Additional patches are
>   needed until QEMU supports a couple additional features that are
>   mandatory for RME (PMUv3p7 and ECV).
>
> Repo: currently at https://jpbrucker.net/git/rmm/ branch qemu-rme
>       official is https://git.trustedfirmware.org/TF-RMM/tf-rmm.git/
>
> Build:  git submodule update --init --recursive
>         export CROSS_COMPILE=aarch64-linux-gnu-
>         cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg 
> -DRMM_TOOLCHAIN=llvm -B build-qemu
>         cmake --build build-qemu
>

The -DRMM_TOOLCHAIN part puzzled me since it was not in the RMM build
instruction for the FVP model.  Is it mandatory and if so, what
version of llvm are you using?  As with the FVP model, I used the
following:

CROSS_COMPILE=/home/mpoirier/work/cca/optee-cca/toolchains/aarch64-none-elf/bin/aarch64-none-elf-
cmake -DRMM_CONFIG=qemu_virt_defcfg -DCMAKE_BUILD_TYPE=Debug
-DLOG_LEVEL=40 -B build
cmake --build build

The "aarch64-none-elf" is exactly the same as the one in the FVP model.

>
> Host EDK2
> ---------
>
> Works out of the box, although it doesn't properly account for reserved
> RMM memory yet.
>

Not sure what to make of the above statement - should I expect the
host boot stack to crash when launching QEMU?

> Repo: https://github.com/tianocore/edk2.git
>
> Build:  git submodule update --init --recursive
>         source edksetup.sh
>         make -j -C BaseTools
>         export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
>         build -b RELEASE -a AARCH64 -t GCC5 -p 
> ArmVirtPkg/ArmVirtQemuKernel.dsc
>

This compiled properly.

>
> TF-A
> ----
>
> TF-A loads the RMM as well as the Non-secure firmware, and bridges RMM and
> KVM. It also owns the Granule Protection Table (GPT).
>
> Status: QEMU support is currently under review.
>
> Repo: currently at https://jpbrucker.net/git/tf-a/ branch qemu-rme
>       official is https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git/
>
> Build:  # Embed the RMM image and edk2 into the Final Image Package (FIP)
>         make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu
>                 ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40
>                 RMM=rmm/build-qemu/Debug/rmm.img
>                 
> BL33=edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
>                 all fip
>
>         # Pack whole image into flash.bin
>         dd if=tfa/build/qemu/debug/bl1.bin of=flash.bin
>         dd if=tfa/build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
>

This compiled properly.

>
> Host & Guest Linux
> ------------------
>
> Both host and guest need extra patches
>
> Status: 
> https://lore.kernel.org/linux-arm-kernel/20231002124311.204614-1-suzuki.poulose@arm.com/
>
> Repo: https://gitlab.arm.com/linux-arm/linux-cca cca-full/rmm-v1.0-eac2
>
> Build: arm64 defconfig
>
>

This compiled properly.

> VMM
> ---
>
> Kvmtool and QEMU can be used to launch Realm guests. For kvmtool, see the
> cover letter for the Linux support above.
>
> Status: In development
>
> Repo: for now https://jpbrucker.net/git/qemu/ branch cca/rmm-v1.0-eac2
>
> Build:  git submodule update --init --recursive
>         # to download the missing subprojects because buildroot won't do it:
>         ./configure --target-list=aarch64-softmmu
>         # Note: when buildroot is configured to use this tree (see below),
>         # it will rsync the whole tree including binary files. Therefore,
>         # use a separate build directory to make the system emulation
>         # QEMU, to avoid conflicts between native binary files and
>         # buildroot's cross-build.
>

This got a little tricky.  From what I understand this is the
"emulation" QEMU, which should likely be in a directory called
"qemu.host".  The QEMU referenced from buildroot should be in another
directory, something like "qemu.buildroot", for buildroot to build.
Is this correct?

>
> Rootfs
> ------
>
> Buildroot provides a convenient way to build lightweight root filesystems.
> It can also embed the VMM into the rootfs if you specify the path to
> kvmtool or QEMU source in a local.mk file in the build directory.
>
> Repo: https://gitlab.com/buildroot.org/buildroot.git
>

Did you use the master branch or a stable one?  If so, which one?

> local.mk
>         KVMTOOL_OVERRIDE_SRCDIR = path/to/kvmtool/
>         QEMU_OVERRIDE_SRCDIR = path/to/qemu/
>

Please confirm that "local.mk" is not part of the buildroot repository
and needs to be created.  Also confirm that KVMTOOLS don't need to be
specified, this is for demonstration purposes.  And following my
comment above, "path/to/qemu" should be "path/to/qemu.buildroot".

> Build:  make qemu_aarch64_virt_defconfig
>         # then change some options
>           # BR2_LINUX_KERNEL is not set
>           BR2_PACKAGE_KVMTOOL=y
>           BR2_PACKAGE_QEMU=y
>           BR2_PACKAGE_QEMU_SYSTEM=y
>           # BR2_PACKAGE_QEMU_BLOBS is not set
>           BR2_PACKAGE_QEMU_SLIRP=y
>           BR2_PACKAGE_QEMU_CHOOSE_TARGETS=y
>           BR2_PACKAGE_QEMU_TARGET_AARCH64=y
>           BR2_TARGET_ROOTFS_EXT2_SIZE=256M
>

I used branch 2023.08.x and several of the above options were _not_
present in the ".config" file, namely BR2_PACKAGE_QEMU_SYSTEM,
BR2_PACKAGE_QEMU_BLOBS, BR2_PACKAGE_QEMU_SLIRP,
BR2_PACKAGE_QEMU_CHOOSE_TARGETS and BR2_PACKAGE_QEMU_TARGET_AARCH64.
Moreover, I tried building buildroot without any changes to the
".config" file and the compilation failed.  This leads me to beleive
we are working on different branches.  I will try the master branch
once I am done with this email.

Clarifications on the above points would be great.

Thanks,
Mathieu

>           # Generate an initrd for the guest
>           BR2_TARGET_ROOTFS_CPIO=y
>
>         make
>
>
> QEMU system emulation
> ---------------------
>
> Repo: https://gitlab.com/qemu-project/qemu.git
>
> Run with:
>
>         qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3
>                 -M acpi=off -cpu max,x-rme=on,sme=off -m 3G -smp 8
>                 -nographic
>                 -bios tf-a/flash.bin
>                 -kernel linux/arch/arm64/boot/Image
>                 -drive 
> format=raw,if=none,file=buildroot/output/images/rootfs.ext4,id=hd0
>                 -device virtio-blk-pci,drive=hd0
>                 -append root=/dev/vda
>
> To use separate consoles for FW (port 54320), secure payload (54321), host
> (54322) and guest (54323):
>
>                 -nodefaults
>                 -serial tcp:localhost:54320
>                 -serial tcp:localhost:54321
>                 -chardev socket,mux=on,id=hvc0,port=54322,host=localhost
>                 -device virtio-serial-device
>                 -device virtconsole,chardev=hvc0
>                 -chardev socket,mux=on,id=hvc1,port=54323,host=localhost
>                 -device virtio-serial-device
>                 -device virtconsole,chardev=hvc1
>                 -append "root=/dev/vda console=hvc0"
>                 -device virtio-net-pci,netdev=net0 -net user,id=net0
>
>   And connect with 'socat -,rawer TCP-LISTEN:5432x'
>
> To share the files needed for booting the guest:
>
>                 -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
>                 -fsdev local,security_model=none,path=.,id=shr0
>
>   And 'mount -t 9p shr0 /mnt' in the host.
>
>
> Then in the host, launch a Realm guest:
>
>         qemu-system-aarch64 -M virt,acpi=off,gic-version=3 -nographic
>                 -cpu host -enable-kvm -smp 2 -m 512M -overcommit mem-lock=on
>                 -kernel /mnt/linux/arch/arm64/boot/Image
>                 -initrd /mnt/buildroot/output/images/rootfs.cpio
>                 -append "console=hvc0 root=/dev/vda"
>                 -M confidential-guest-support=rme0
>                 -object 
> rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256
>                 -nodefaults -chardev stdio,mux=on,id=hvc0,signal=off
>                 -device virtio-serial-pci -device virtconsole,chardev=hvc0
>                 -device virtio-net-pci,netdev=net0,romfile= -netdev 
> user,id=net0
>                 </dev/hvc1 >/dev/hvc1 &
>         ...
>         [    0.000000] RME: RSI version 13.0 advertised
>