Re: [Qemu-discuss] Debugging qemu inside chroot

[Paulo Matos]

On 28/03/2019 11:20, Peter Maydell wrote:
>>
>> Thanks for your help.
>> At this point I get this from debugging qemu-aarch64-static:
>> process 4399 is executing new program:
>> /root/aarch64-chroot/usr/bin/qemu-aarch64-static
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> [New Thread 0x7ffff7ff9700 (LWP 4403)]
>>
>> Thread 1 "mksystem.rkt" received signal SIGSEGV, Segmentation fault.
>> 0x0000000060b6f9a9 in static_code_gen_buffer ()
>> (gdb) bt
>> #0  0x0000000060b6f9a9 in static_code_gen_buffer ()
>> #1  0x000000006004abc4 in cpu_tb_exec (cpu=0x6296ee30,
>>     itb=0x60b6f8c0 <static_code_gen_buffer+2357056>)
>>     at /root/qemu-3.1.0/accel/tcg/cpu-exec.c:171
>> #2  0x000000006004b807 in cpu_loop_exec_tb (cpu=0x6296ee30,
>>     tb=0x60b6f8c0 <static_code_gen_buffer+2357056>, last_tb=0x7fffffffdbd8,
>>     tb_exit=0x7fffffffdbd0) at /root/qemu-3.1.0/accel/tcg/cpu-exec.c:615
>> #3  0x000000006004baa0 in cpu_exec (cpu=0x6296ee30)
>>     at /root/qemu-3.1.0/accel/tcg/cpu-exec.c:725
>> #4  0x0000000060084eef in cpu_loop (env=0x629770e0)
>>     at /root/qemu-3.1.0/linux-user/aarch64/cpu_loop.c:82
>> #5  0x000000006005a042 in main (argc=10, argv=0x7fffffffe508,
>> envp=0x7fffffffe560)
>>     at /root/qemu-3.1.0/linux-user/main.c:819
> 
> This backtrace says "the segfault happened in code that
> QEMU generated" (because the faulting PC is in 'static_code_gen_buffer',
> which is where we put the x86 code we generate for the guest
> binary). This usually means "it's a segfault in the guest".
> 
> I think at this point I would take the "use QEMU's gdbstub
> and an aarch64-aware debugger to try to debug the guest"
> and see if the backtrace and information you get there
> suggest what's happening.
> 

I was debugging qemu because I had previously tested debugging the
racket process unsuccessfully.

I found out why previously my attempts were unsuccessful. I forgot to
set sysroot and I got:
warning: remote target does not support file transfer, attempting to
access files from local filesystem.
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Now I set sysroot but still having no luck. On one session:

# chroot aarch64-chroot/ qemu-aarch64-static -g 1234
/racket/racket/src/build/racket/racket3m ...


Then on another session:

# gdb-multiarch GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) set arch aarch64
The target architecture is assumed to be aarch64
(gdb) set sysroot aarch64-chroot/
(gdb) file aarch64-chroot/racket/racket/src/build/racket/racket3m
Reading symbols from
aarch64-chroot/racket/racket/src/build/racket/racket3m...done.
(gdb) target remote :1234
Remote debugging using :1234
Reading symbols from aarch64-chroot/lib/ld-linux-aarch64.so.1...(no
debugging symbols found)...done.
0x0000004000fe9cc0 in ?? () from aarch64-chroot/lib/ld-linux-aarch64.so.1
(gdb) handle SIGSEGV noprint
Signal        Stop      Print   Pass to program Description
SIGSEGV       No        No      Yes             Segmentation fault
(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
0x00000040011309fc in ?? ()
(gdb) bt
#0  0x00000040011309fc in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

The reason I noprint SIGSEGV is because the racket garbage collector
uses these and we need to pass them but not stop if we want to debug
racket. If I don't do this I don't get to main either. Restarting gdb
and with the same gdb prologue commands as before (without continuing on
sigsegv) I am still getting no information.

(gdb) b main
Breakpoint 1 at 0x2c6e4: file ../../../racket/gc2/../main.c, line 326.
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000004000095484 in ?? ()
(gdb) bt
#0  0x0000004000095484 in ?? ()
#1  0x00000040006f6734 in ?? ()
#2  0x000000400124c260 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind
further

If I look in the disassembly for instruction at address 95484 I end up
in scheme_gmp_tls_unload.

   95474:       b00038c0        adrp    x0, 7ae000
<mapped_uchar_ranges+0x968>
   95478:       913e8000        add     x0, x0, #0xfa0
   9547c:       f9000001        str     x1, [x0]
   95480:       f94007e0        ldr     x0, [sp, #8]
   95484:       f900001f        str     xzr, [x0]      ; <<<==========
   95488:       b00038c0        adrp    x0, 7ae000
<mapped_uchar_ranges+0x968>
   9548c:       913dc000        add     x0, x0, #0xf70
   95490:       f94003e1        ldr     x1, [sp]
   95494:       f9000001        str     x1, [x0]
   95498:       d503201f        nop
   9549c:       910043ff        add     sp, sp, #0x10
   954a0:       d65f03c0        ret


But if I set a breakpoint in that function, it doesn't even stop.

(gdb) set arch aarch64
The target architecture is assumed to be aarch64
(gdb) set sysroot aarch64-chroot/
(gdb) file aarch64-chroot/racket/racket/src/build/racket/racket3m
Reading symbols from
aarch64-chroot/racket/racket/src/build/racket/racket3m...done.
(gdb) target remote :1234
Remote debugging using :1234
Reading symbols from aarch64-chroot/lib/ld-linux-aarch64.so.1...(no
debugging symbols found)...done.
0x0000004000fe9cc0 in ?? () from aarch64-chroot/lib/ld-linux-aarch64.so.1
(gdb) b scheme_gmp_tls_unload
Breakpoint 1 at 0x95430: file ../../../racket/gc2/../src/gmp/gmp.c, line
5819.
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000004000095484 in ?? ()
(gdb) bt
#0  0x0000004000095484 in ?? ()
#1  0x00000040006f6734 in ?? ()
#2  0x000000400124c260 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind
further



I don't even understand if I am still in the realm of a qemu problem, so
this might be getting off-topic. I just wanted to confirm with you that
I am not missing anything (like I initially missed `set sysroot`).

> 
> Yes. (In other words, we do just-in-time translation -- JITting --
> of the guest binary.)

>> So above, qemu crashes at PC 0x40011309e0 which corresponds to guest:
>> mov      w4, w0
> 
> No, this isn't correct. You can from the gdb backtrace that
> QEMU crashed at host PC 0x0000000060b6f9a9, which is not in
> the translation block that you quote above. The TB we were
> in when we crashed will have been translated a bit further up
> in the logs, I expect.
> 
> QEMU caches translation blocks, so the last block translated
> is not necessarily the last block executed. You can get QEMU
> to tell you what blocks it is executing by adding 'cpu,exec'
> to the set of -d flags, but this will make execution rather
> slower. I think this is likely to be a more longwinded way of
> debugging further than using the QEMU gdbstub.
> 
>> At this point I should mention I don't know much about aarch64 but I
>> didn't think it had wX registers.
> 
> In AArch64, wN is "the bottom 32 bits of register xN". So
> mov w4, w0 is doing a 32 bit move (whereas mov x4, x0 would be
> the 64 bit move) that will clear the top 32 bits of the destination
> register.
> 

Thanks a lot for your time clarifying this.

-- 
Paulo Matos