Re: Quick start guide on RME emulation

[Jean-Philippe Brucker]

On Tue, Oct 17, 2023 at 04:39:08PM -0600, Mathieu Poirier wrote:
> On Tue, 17 Oct 2023 at 07:56, Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > On Mon, Oct 16, 2023 at 03:30:30PM -0600, Mathieu Poirier wrote:
> > > > TF-RMM
> > > > ------
> > > >
> > > > The Realm Management Monitor (RMM) connects KVM and the Realm guest.
> > > >
> > > > RMM gets loaded into NS DRAM (because there isn't enough space in Secure
> > > > RAM). TF-A carves out 24MB of memory for the RMM, and tells other 
> > > > software
> > > > about it using a device-tree reserved memory node.
> > > >
> > > > Status: QEMU support is currently under review. Additional patches are
> > > >   needed until QEMU supports a couple additional features that are
> > > >   mandatory for RME (PMUv3p7 and ECV).
> > > >
> > > > Repo: currently at https://jpbrucker.net/git/rmm/ branch qemu-rme
> > > >       official is https://git.trustedfirmware.org/TF-RMM/tf-rmm.git/
> > > >
> > > > Build:  git submodule update --init --recursive
> > > >         export CROSS_COMPILE=aarch64-linux-gnu-
> > > >         cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg 
> > > > -DRMM_TOOLCHAIN=llvm -B build-qemu
> > > >         cmake --build build-qemu
> > > >
> > >
> > > The -DRMM_TOOLCHAIN part puzzled me since it was not in the RMM build
> > > instruction for the FVP model.  Is it mandatory and if so, what
> > > version of llvm are you using?  As with the FVP model, I used the
> > > following:
> > >
> > > CROSS_COMPILE=/home/mpoirier/work/cca/optee-cca/toolchains/aarch64-none-elf/bin/aarch64-none-elf-
> > > cmake -DRMM_CONFIG=qemu_virt_defcfg -DCMAKE_BUILD_TYPE=Debug
> > > -DLOG_LEVEL=40 -B build
> > > cmake --build build
> > >
> > > The "aarch64-none-elf" is exactly the same as the one in the FVP model.
> >
> > Both GCC and LLVM should work, but I had some trouble with GCC and haven't
> > had time to investigate yet, so I provided instructions for LLVM (v16)
> > which works for me. I'm probably using the wrong GCC toolchain, because
> > I'm currently getting these link errors:
> >
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > warning: libc.so.6, needed by /usr/aarch64-linux-gnu/lib/libm.so.6, not 
> > found (try using -rpath or -rpath-link)
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `__strtold_nan@GLIBC_PRIVATE'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `stderr@GLIBC_2.17'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `qsort@GLIBC_2.17'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `__strtof_nan@GLIBC_PRIVATE'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `fwrite@GLIBC_2.17'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `errno@GLIBC_PRIVATE'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `fputs@GLIBC_2.17'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `__strtod_nan@GLIBC_PRIVATE'
> > /usr/lib/gcc/aarch64-linux-gnu/13.2.0/../../../../aarch64-linux-gnu/bin/ld: 
> > /usr/aarch64-linux-gnu/lib/libm.so.6: undefined reference to 
> > `__assert_fail@GLIBC_2.17'
> >
> 
> I had the same errors, which is why I switched to aarch64-none-elf
> [1], the same as what is used for the FVP code base.  If you have
> time, try that on your side and let me know if things still work.

Nice, I can confirm that switching to aarch64-none-elf fixes the build
issue, and I can boot a realm guest with the RMM built that way. It feels
like some compiler/linker flag could enable building with
aarch64-linux-gnu as well, but I don't have time to look into this at the
moment.

Thanks,
Jean