gdbstub and gbd segfaults on different instructions in user space emulation

[Libo Zhou]

Hi all,

I am encountering segmentation fault while porting my custom ISA to QEMU. My 
custom ISA is VERY VERY simple, it only changes the [31:26] opcode field of LW 
and SW instructions. The link has my very simple implementation: 
https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg06976.html

Below is the objdump of the main part of my simple ELF. The dots are the 
automatically generated nop's by the compiler.
00400090 <main>:
  400090:       23bdffe0    addi    r29,r29,-32
  400094:       7fbe001c    sw    r30,28(r29.
        ...
  4000a0:       03a0f021    addu    r30,r29,r0
  4000a4:       20020001    li    r2,1    # int a = 1;
  4000a8:       7fc20010    sw    r2,16(r30)
        ...
  4000b4:       20020002    li    r2,2    # int b = 2;
  4000b8:       7fc2000c    sw    r2,12(r30)
        ...
  4000c4:       5fc30010    lw    r3,16(r30)
  4000c8:       00000000    nop
  4000cc:       5fc2000c    lw    r2,12(r30)
        ...
  4000d8:       00621020    add    r2,r3,r2    # int c = a + b;
  4000dc:       7fc20008    sw    r2,8(r30)
        ...
  4000e8:       00001021    addu    r2,r0,r0
  4000ec:       03c0e821    addu    r29,r30,r0
  4000f0:       5fbe001c    lw    r30,28(r29)
  4000f4:       23bd0020    addi    r29,r29,32
  4000f8:       03e00008    jr    r31
  4000fc:       00000000    nop
        ...

The code below gives me segfault:
$ ./qemu-mipsel -cpu mycpu testprogram

I have tried 2 ways of debugging it.
Firstly, I connected gdb-multiarch to gdbstub, and I single-stepped the 
instructions in my ELF. Immediately after the LW instruction, the segfault was 
thrown. I observed the memory location using 'x' command and found that at 
least my SW instruction was implemented correctly.
Secondly, I used gdb to directly debug QEMU. I set the breakpoint at function 
in translate.c:decode_opc. Pressing 'c' should have the same effect as 
single-stepping instruction in gdbstub. However, the segmentation fault wasn't 
thrown after LW. It was instead thrown after the 'nop' after 'jr r31' in the 
objdump.

At this point, I am really stuck. I have spent a long time on this, but I just 
can't figure out what is going wrong here. If anyone can help me out I would 
really appreciate it.

Cheers,
Libo Zhou