[Qemu-discuss] Possible Qemu inconsistency when emulating TLB for rpi3 machine

[Mattia Maldini]

Hello everyone,
I'm not sure if I'm in the right place to ask my question, and I don't
usually use mailing list, so it's possible that my message is completely
misplaces. I'm looking for directions anyway, so please do correct my
approach if anything is wrong.

I am developing a sort of hardware abstraction layer to assist
undergraduate students in the task of creating a toy OS for Raspberry Pi 3.
A big part of this project is the possibility to run the OS both on real
hardware and Qemu, which supports the AArch64 rpi3 machine since version
2.12 .

Everything was running smoothly until I started using the MMU: I find
myself in a situation where the same code yields different results on Qemu
and on RPi3, and I believe only the real hardware is behaving correctly. I
set up a very simplified example to showcase my problem.

Basically I have two page tables (identical mapping, just to test the
virtual memory): one for the "kernel" running at EL1 and one for a "user
process" running at EL0.
When the process is running TTBR0 must point to its page table because of
permissions: page entries in the EL0 table must include EL0 read-write
permissions, while page entries in the EL1 table must forbid them. Each
actor (kernel & process) MUST then run on its own set of page tables.

Everything the kernel does is setting up the MMU and launching the process
at EL0; while doing so, he also sets TTBR0 with the correct page table for
the process. Note that at this point page entries are configured as global
and I'm not using any ASID.

This situation works on Qemu but not on real hardware, and I think the
emulator is wrong. Both kernel and process have their code in the same
block of memory, so they share MMU entries. When the kernel starts
executing its code pages are loaded and saved in the TLB; when the process
starts it should ask for the same entries and receive the kernel pages
saved in the TLB, failing because they are configured to prevent EL0
access. This is what happens on the real RPi3.

To solve my problem I started using an ASID for the process: I put some
value into the two most significant bytes of the TTBR0 register and set all
page entries as Non-Global. This way when the process asks for a table
entry the ones from the kernel found in the TLB cache are discarded because
of the difference in ASID.
This solution works as expected on the Raspberry Pi 3 but results in an
abort exception (permission fault) on Qemu.

Furthermore, it seems that Qemu aborts every time it looks for a different
ASID in the TLB (this is a conjecture, I'm not a Qemu developer myself). I
have studied the ARMv8 MMU thoroughly and I think my configuration is
correct.

Despite being confused about the situation, the fact holds that the real
hardware behaves as expected while Qemu does not. What I want to ask at
this point is:
    1. Does Qemu emulate the ASID-TLB interaction for AArch64? Is this
behaviour expected? Do you see an obvious mistake on my side?
    2. Can this be a bug in the rpi3 machine Qemu implementation? How could
I investigate further?

For reference, this is the Github repo that hosts my code so far:
https://github.com/Maldus512/uARM_pienv.git