I can reproduce the problem using rlwrap-0.37 and readline-6.3 (Gentoo). I didn't try using gdb, but (a suitably configured) rlwrap has an option to log everything that happens on stdin (rlwrap -d cat)
Pressing "Delete" sends kdch1 (the delete-character key) in rxvt and xterm, this is "<ESC> [ 3 ~"
Pressing it twice sends "<ESC> [ 3 ~ <ESC> [ 3 ~".
After reading each input character rlwrap calls readine_callback_read_character()
The crash happens in the call after the second <ESC>