QEMU CAS

[Jasper Ruehl]

Dear QEMU Community,

after chatting a bit in the IRC channel, "stefanha" advised me to contact you via EMail about my problem.
My advisor and me from the DSE chair at the TU Munich had the idea to improve the emulation of the x86 cmpxchg instruction on ARM64 CPUs by improving the translation scheme: instead of generating a call to the GCC helper function, we introduce a new TCG IR instruction (CAS) and directly translate that into the ARM casal instruction.

During benchmarking, we noticed that our version of QEMU performs correctly if we disable TCG optimizations (done in tcg/tcg.c:26).
If we enable them, however, the emulated program hangs indefinitely in futex syscalls.
We are not sure if this misbehavior stems from a bug in the implementation or if it is due to some assumptions made by the optimizer.

We have tried to analyse the program using -d in_asm,op,op_opt,out_asm, but could not determine the issue.

Using gdb, we could determine where the threads hang.

Threads 3 - END are in the futex syscall of the function do_futex_wait, waiting for thread 2 to be created.
Judging by the stack trace, Thread 2 seems to still be in the creation routine:

#0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1  0x0000aaaad21b4e60 in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at /qemu/qemu/include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0xaaaad25a9470 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:480
#3  0x0000aaaad21be044 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:258
#4  0x0000aaaad21b3d38 in qemu_thread_start (args=<optimized out>) at ../util/qemu-thread-posix.c:541
#5  0x0000ffffb2626f5c in start_thread (arg=0x0) at pthread_create.c:463
#6  0x0000ffffb257eb1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

The code for the creation of the threads is found at phoenix2.0/src/tpool.c:tpool_create
Thread 2 is created the same way the others are, however it is the only one failing.

Commit introducing the native CAS emulation: https://github.com/rgouicem/qemu/commit/5bc56e203936338d98acdb868786834c751f87a7

Repo with the code demonstrating the problem: https://github.com/haxkor/qemu_phoenix
Run it with qemu_opt ./histogram small.bmp

I have included my binary and the source code if you want to build it yourself, simply run "make" in the phoenix2 folder.
The binary will be at phoenix2/tests/histogram/histogram

We hope to have provided you useful information. Please let us know if there is anything else we can do.

Best regards,
Redha Gouicem and Jasper Ruehl.