Re: Calling readline() a second or subsequent time and retaining state from previous calls

[Sam Habiel]

Thank you Chet.

I proceeded with implementing something that is split up across calls to 
readline. It's not perfect, and I am writing to you about the problems I faced. 
I totally understand if you think it's crazy and think I should abandon the 
effort, but I mostly got it to work.

The current source code is here: 
https://gitlab.com/shabiel/YDB/-/blob/ydb83_readline/sr_unix/readline.c (it 
will change a bit in the next few days, but not materially, so you can still 
refer to it).

My approach can be summarized as follows:

When I receive a SIGUSR1, siglongjmp to code that does the following:
- save the current state using rl_save_state
- Copy out the state->buffer and state->prompt
- NULL out rl_prompt (otherwise, it gets freed by readline on the next call)
- free state->buffer
- free state->prompt
- Put my buffer and prompt back into the state structure (but I did the 
allocation and they won't disappear)
- unwind the C stack
- Call readline again with the following hooks
-- startup hook: restore state using rl_restore_state, and set 
rl_already_prompted to 1.
-- redisplay hook: no-op: The text is already on the screen and we don't want 
it displayed again.

I compiled readline from master with -g -O3 so I can see what's going on.

Here are the two major issues I faced:
1. I want to call rl_free_line_state after saving state, but doing so, after 
restoring state, I get a double free warning from ASAN. The undo list is 
attempted to be freed again:

==68415==ERROR: AddressSanitizer: attempting double-free on 0x603000001ae0 in 
thread T0:
    #0 0x7f5e7c27edb8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdbdb8)
    #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111
    #2 0x7f5e763cb178 in rl_free_undo_list ../undo.c:122
    #3 0x7f5e763ac32c in readline_internal_teardown ../readline.c:512
    #4 0x7f5e763ada1a in readline_internal ../readline.c:734
    #5 0x7f5e763ada1a in readline ../readline.c:387
    #6 0x7f5e76d2c7ef in readline_read_mval 
/home/sam/work/gitlab/YDB/sr_unix/readline.c:428
    #7 0x7f5e77fa53b7 in op_dmode 
/home/sam/work/gitlab/YDB/sr_port/op_dmode.c:126
    #8 0x7f5e776f0c50  (/home/sam/work/builds/YDB/libyottadb.so+0xfaac50)

0x603000001ae0 is located 0 bytes inside of 32-byte region 
[0x603000001ae0,0x603000001b00)
freed by thread T0 here:
    #0 0x7f5e7c27edb8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdbdb8)
    #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111

previously allocated by thread T0 here:
    #0 0x7f5e7c27f110 in malloc (/usr/lib64/libasan.so.4+0xdc110)

2. If I bypass the readline redisplay, I cannot see any characters being typed. 
I tried calling rl_reset_terminal and rl_tty_set_echoing to no avail. I think 
it has to do with internal readline state (maybe it thinks my cursor is 
somewhere else), but the re-display code is so complex that it's hard for me to 
tell.

Let me know if you have any advice; and if not, that's fine too... I do know 
what I am doing is crazy, and maybe when we switch in the future to the 
callback interface, things would be easier and I won't need all of these 
shenanigans.

--Sam


 ---- On Mon, 25 Sep 2023 09:30:44 -0400  Chet Ramey  wrote --- 
 > On 9/21/23 12:34 PM, Sam Habiel wrote:
 > > Hello there,
 > > 
 > > Quick Intro: We have an interactive CLI for a language called M used in 
 > > banking and healthcare... I am working on adding readline integration as a 
 > > nice to have feature. We are an open source project located over here: 
 > > https://gitlab.com/YottaDB/DB/YDB.
 > > 
 > > I sent an earlier message, and I thank Chet for the reply.
 > > 
 > > I am dealing with the unpleasant for anybody to handle signals. One of our 
 > > signals suspends execution, completely unwinds the C stack (yes, we have 
 > > lots of assembly code for that), and then reissues the prompt. An example 
 > > will help:
 > > 
 > > YDB>write "fo  <<< Receive SIGUSR1
 > > 
 > > Once a SIGUSR1 is received, in the background, execution is suspended, a 
 > > custom function is executed, and the entire prompt code is actually 
 > > processed, but the user sees absolutely nothing of that. In the example 
 > > above, the cursor will stay between f and o. Our current code just sets up 
 > > all the prompt data structures from a saved structure when the signal is 
 > > received.
 > > 
 > > Is this possible with readline()? I naively tried rl_save_state() and 
 > > rl_restore_state(), but that doesn't seem to work.
 > > 
 > 
 > SIGUSR1 isn't one of the signals readline handles by default, but it has
 > enough API functions available so that you can do it in your application.
 > 
 > All the necessary functions are in signals.c. The basic idea is that you
 > install your signal handler as usual, call the readline functions you
 > want to restore the terminal and anything else from the handler when it's
 > safe (look at _rl_signal_handler() for examples), then restore before your
 > handler returns (look at rl_cleanup_after_signal/rl_restore_after_signal).
 > 
 > If you want to split it up across calls to readline, you'll have to save
 > and restore a lot of state. I don't know if the information in
 > struct readline_state is enough for you -- it depends on your application
 > needs -- and it's not really intended to save and restore across calls
 > to readline(). It's mostly intended as a checkpoint during a single call,
 > so the interaction between the initialization readline does on each
 > invocation and the state restored isn't well exercised.
 > 
 > If you want to go further with that approach, you'll need at least a
 > startup hook to restore the state and do a redisplay before going on
 > in the second (post-signal) call to readline.
 > 
 > Chet
 > 
 > -- 
 > ``The lyf so short, the craft so long to lerne.'' - Chaucer
 >          ``Ars longa, vita brevis'' - Hippocrates
 > Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/
 > 
 >