Re: could use some help debugging an unexpected emulator exit

[Peter Maydell]

On Wed, 22 Jan 2020 at 18:09, Andrew Boie <address@hidden> wrote:
>
> I'm working on the x86-64 port of a microcontroller RTOS
> (https://www.zephyrproject.org/ if anyone's interested) and am running
> into a situation where QEMU appears to be suddenly exiting without
> printing any debug information. What I'm pretty sure is happening is a
> triple fault, I've seen other situations where QEMU just quits when a
> triple fault happens, or I have programmed bad page tables, etc.
>
> What I'd like is some advice for dumping out the register state when
> something like this happens. I did manage to reproduce this under GDB
> and got the following stack trace with QEMU 4.2.0. My knowledge of
> QEMU's internals is poor, but it looks like we've got a write to a
> NULL address:
>
> Thread 3 "qemu-system-x86" hit Breakpoint 1, __GI_exit (status=1) at 
> exit.c:138
> 138    exit.c: No such file or directory.
> (gdb) bt
> #0  0x00007ffff52a1d40 in __GI_exit (status=1) at exit.c:138
> #1  0x0000555555aa5eb3 in debug_exit_write
>     (opaque=<optimized out>, addr=<optimized out>, val=<optimized
> out>, width=<optimized out>)
>     at hw/misc/debugexit.c:35
> #2  0x000055555589313b in memory_region_write_accessor
>     (mr=mr@entry=0x5555565ac490, addr=0,
> value=value@entry=0x7fffe8c07e98, size=size@entry=4, shift=)
>     at /home/apboie/Downloads/qemu-4.2.0/memory.c:483

What's happening here is that the board you're running on includes
the "debug exit port" device. The purpose of this device is to
silently exit with an exit status based on the value written to
the device, if the guest code writes to it. Your guest code
wrote to the device, and QEMU exited as requested :-)

More specifically:
 * this device is optional (if I'm reading the code right), so
you can get rid of it by editing your QEMU command line to remove it
(there should be a '-device isa-debug-exit,...')
 * by default it's at x86 I/O port 0x501, but the port number
can be changed by command line option so it might not be there
for you
 * since guests don't usually do random bogus I/O port accesses,
my first guess would be that your RTOS is deliberately exiting.

Handily, since you're using an open source RTOS I can just
look at the code, and I think the chances are about 99%
that it has called arch_system_halt():

https://github.com/zephyrproject-rtos/zephyr/blob/2690c9e550024f13dec6f6df7c9fe6046b67ee70/arch/x86/core/fatal.c#L15

Ways to get your register state:
 * make zephyr do it in arch_system_halt(), eg by
calling dump_state() -- this will also get you a stack
trace I think
 * make zephyr's arch_system_halt() instead do an
int3 breakpoint insn, and run with a debugger
attached to the QEMU gdbstub (ie debug the guest
under gdb, not QEMU itself)
 * if the runtime is short, you can use the various
suboptions under the QEMU -d option: 'cpu' will print
the CPU state before every executed block of translated
code, so the output will be very large for a long run,
but for short "crashes immediately" tests this logging
can be helpful.
 * you can hack QEMU to make hw/misc/debugexit.c:debug_exit_write()
call hw_error("Guest wrote to debug exit port\n");
which will make QEMU print a guest register dump and abort().

thanks
-- PMM