Re: Quick start guide on RME emulation

[Mathieu Poirier]

On Tue, 17 Oct 2023 at 07:56, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Oct 16, 2023 at 03:30:30PM -0600, Mathieu Poirier wrote:
> > > TF-RMM
> > > ------
> > >
> > > The Realm Management Monitor (RMM) connects KVM and the Realm guest.
> > >
> > > RMM gets loaded into NS DRAM (because there isn't enough space in Secure
> > > RAM). TF-A carves out 24MB of memory for the RMM, and tells other software
> > > about it using a device-tree reserved memory node.
> > >
> > > Status: QEMU support is currently under review. Additional patches are
> > >   needed until QEMU supports a couple additional features that are
> > >   mandatory for RME (PMUv3p7 and ECV).
> > >
> > > Repo: currently at https://jpbrucker.net/git/rmm/ branch qemu-rme
> > >       official is https://git.trustedfirmware.org/TF-RMM/tf-rmm.git/
> > >
> > > Build:  git submodule update --init --recursive
> > >         export CROSS_COMPILE=aarch64-linux-gnu-
> > >         cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg 
> > > -DRMM_TOOLCHAIN=llvm -B build-qemu
> > >         cmake --build build-qemu
> > >
> >
> > The -DRMM_TOOLCHAIN part puzzled me since it was not in the RMM build
> > instruction for the FVP model.  Is it mandatory and if so, what
> > version of llvm are you using?  As with the FVP model, I used the
> > following:
> >
> > CROSS_COMPILE=/home/mpoirier/work/cca/optee-cca/toolchains/aarch64-none-elf/bin/aarch64-none-elf-
> > cmake -DRMM_CONFIG=qemu_virt_defcfg -DCMAKE_BUILD_TYPE=Debug
> > -DLOG_LEVEL=40 -B build
> > cmake --build build
> >
> > The "aarch64-none-elf" is exactly the same as the one in the FVP model.
>
> Both GCC and LLVM should work, but I had some trouble with GCC and haven't
> had time to investigate yet, so I provided instructions for LLVM (v16)
> which works for me. I'm probably using the wrong GCC toolchain, because
> I'm currently getting these link errors:
>
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> warning: libc.so.6, needed by /usr/aarch64-linux-gnu/lib/libm.so.6, not found 
> (try using -rpath or -rpath-link)
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `__strtold_nan@GLIBC_PRIVATE'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `stderr@GLIBC_2.17'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `qsort@GLIBC_2.17'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `__strtof_nan@GLIBC_PRIVATE'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `fwrite@GLIBC_2.17'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `errno@GLIBC_PRIVATE'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `fputs@GLIBC_2.17'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `__strtod_nan@GLIBC_PRIVATE'
> /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> `__assert_fail@GLIBC_2.17'
>

I had the same errors, which is why I switched to aarch64-none-elf
[1], the same as what is used for the FVP code base.  If you have
time, try that on your side and let me know if things still work.

[1]. 
https://developer.arm.com/-/media/Files/downloads/gnu/12.3.rel1/binrel/arm-gnu-toolchain-12.3.rel1-x86_64-aarch64-none-elf.tar.xz?rev=a8bbb76353aa44a69ce6b11fd560142d&hash=20124930455F791137DDEA1F0AF79B10

>
> >
> > >
> > > Host EDK2
> > > ---------
> > >
> > > Works out of the box, although it doesn't properly account for reserved
> > > RMM memory yet.
> > >
> >
> > Not sure what to make of the above statement - should I expect the
> > host boot stack to crash when launching QEMU?
>
> I'm still figuring that out but I don't think so, because edk2 seems to
> allocate stack and heap memory from the top of RAM, so it won't touch the
> RMM memory (which I placed near the bottom at 0x40100000-0x418fffff).
>
> But we can't rely on this, so long term we'll need to make edk2 aware that
> there is a hole in RAM. I need a better understanding of edk2 memory
> management first.
>

Ok, I hope to get to that part tomorrow.

> > > VMM
> > > ---
> > >
> > > Kvmtool and QEMU can be used to launch Realm guests. For kvmtool, see the
> > > cover letter for the Linux support above.
> > >
> > > Status: In development
> > >
> > > Repo: for now https://jpbrucker.net/git/qemu/ branch cca/rmm-v1.0-eac2
> > >
> > > Build:  git submodule update --init --recursive
> > >         # to download the missing subprojects because buildroot won't do 
> > > it:
> > >         ./configure --target-list=aarch64-softmmu
> > >         # Note: when buildroot is configured to use this tree (see below),
> > >         # it will rsync the whole tree including binary files. Therefore,
> > >         # use a separate build directory to make the system emulation
> > >         # QEMU, to avoid conflicts between native binary files and
> > >         # buildroot's cross-build.
> > >
> >
> > This got a little tricky.  From what I understand this is the
> > "emulation" QEMU, which should likely be in a directory called
> > "qemu.host".  The QEMU referenced from buildroot should be in another
> > directory, something like "qemu.buildroot", for buildroot to build.
> > Is this correct?
>
> Yes. Two separate source trees should be easiest to deal with.
>
> To ensure my VMM changes don't break emulation QEMU I use a single source
> tree, and a separate build directory for the emulation QEMU:
>
>         cd ~/build/qemu.host/
>         ~/src/qemu/configure --target-list=aarch64-softmmu
>         make -j
>
> > >
> > > Rootfs
> > > ------
> > >
> > > Buildroot provides a convenient way to build lightweight root filesystems.
> > > It can also embed the VMM into the rootfs if you specify the path to
> > > kvmtool or QEMU source in a local.mk file in the build directory.
> > >
> > > Repo: https://gitlab.com/buildroot.org/buildroot.git
> > >
> >
> > Did you use the master branch or a stable one?  If so, which one?
>
> I'm using the master branch (currently I'm on 96130dc204)
>
> >
> > > local.mk
> > >         KVMTOOL_OVERRIDE_SRCDIR = path/to/kvmtool/
> > >         QEMU_OVERRIDE_SRCDIR = path/to/qemu/
> > >
> >
> > Please confirm that "local.mk" is not part of the buildroot repository
> > and needs to be created.  Also confirm that KVMTOOLS don't need to be
> > specified, this is for demonstration purposes.  And following my
> > comment above, "path/to/qemu" should be "path/to/qemu.buildroot".
>
> Yes, local.mk needs to be created, and KVMTOOL_OVERRIDE_SRCDIR isn't
> needed if you're planning to only use QEMU to launch Realm VMs
>
> >
> > > Build:  make qemu_aarch64_virt_defconfig
> > >         # then change some options
> > >           # BR2_LINUX_KERNEL is not set
> > >           BR2_PACKAGE_KVMTOOL=y
> > >           BR2_PACKAGE_QEMU=y
> > >           BR2_PACKAGE_QEMU_SYSTEM=y
> > >           # BR2_PACKAGE_QEMU_BLOBS is not set
> > >           BR2_PACKAGE_QEMU_SLIRP=y
> > >           BR2_PACKAGE_QEMU_CHOOSE_TARGETS=y
> > >           BR2_PACKAGE_QEMU_TARGET_AARCH64=y
> > >           BR2_TARGET_ROOTFS_EXT2_SIZE=256M
> > >
> >
> > I used branch 2023.08.x and several of the above options were _not_
> > present in the ".config" file, namely BR2_PACKAGE_QEMU_SYSTEM,
> > BR2_PACKAGE_QEMU_BLOBS, BR2_PACKAGE_QEMU_SLIRP,
> > BR2_PACKAGE_QEMU_CHOOSE_TARGETS and BR2_PACKAGE_QEMU_TARGET_AARCH64.
> > Moreover, I tried building buildroot without any changes to the
> > ".config" file and the compilation failed.  This leads me to beleive
> > we are working on different branches.  I will try the master branch
> > once I am done with this email.
> >
>

I figured that part - missing options are because
qemu_aarch64_virt_defconfig doesn't include QEMU, it has to be
enabled.  To do so "make menuconfig" needs to be used rather than
adding the configuration by hand, otherwise some options will be
missing.

> Buildroot keeps the QEMU build recipe in package/qemu/qemu.mk which only
> has instructions for one version of QEMU (QEMU_VERSION). Since QEMU's
> configure options change regularly and my VMM modifications are based on
> QEMU's master branch, I use buildroot's master branch in order to get the
> most recent qemu.mk.
>

I also figured that problem out - have a look at this [2].  Simply
put, that patch moves the handling of submodules to the mason build
system.  As such doing "git submodule update --init --recursive" won't
work to download the submodules in the QEMU code base.  Nowadays QEMU
submodules are downloaded as part of the configure script that calls
mason.  So calling "./configure --target-list=aarch64-softmmu" will
download the submodules.  So the procedure becomes:

$ cd qemu.target
$  ./configure --target-list=aarch64-softmmu //downloads the
submodules as part of the configuration process
$ make clean  //removes the "build" directory and configuration files,
but leaves the submodules.
$ // qemu.target is now ready to be referenced in the local.mk file in
the buildroot directory

The above is a hack to deal with an upstream regression.  We will have
to address that regression but for now all I want is to reproduce your
environment, something I will hopefully be able to achieve tomorrow.

[2].  https://www.mail-archive.com/qemu-devel@nongnu.org/msg967757.html

> Thanks,
> Jean