[Bug-readline] immediate acceptance of ESC in vi mode

[Barry Downes]

Short version:
I had a problem with bash/readline misinterpreting input sequences in vi 
mode and erasing half my input line.  To fix it I got my terminal to 
send the less ambiguous "\eO[" instead of just "\e" for the escape key.  
But there was a hack in the readline code which was still causing 
problems, so I added the "vi-escape-hack" variable so I could disable 
the hack from my .inputrc.  The patch for this variable is attached.  
Naturally readline's behaviour remains unchanged for the default value 
of the variable.

Long version:
I had a problem with the arrow/movement keys causing half my line to be 
wiped out in the vi insert mode of readline, which I experienced when 
using bash and mysql.  It happened on average about once for an entire 
movement from the left to the right edge of my window, by holding the 
arrow key and letting it repeat.

I first encountered the problem with bash 4.0.38:
$ uname -srvmo
Linux 2.6.32.23-170.fc12.x86_64 #1 SMP Mon Sep 27 17:23:59 UTC 2010 
x86_64 GNU/Linux
$ /bin/bash --version
GNU bash, version 4.0.38(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ ldd /bin/bash | grep readline

Then I built bash 4.2.0 from source to use an external readline 6.2:
$ ~/bin/bash --version
GNU bash, version 4.2.0(2)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ ldd ~/bin/bash | grep readline
        libreadline.so.6 => /home/barry/fakeroot/lib/libreadline.so.6 
(0x00007ff33e1a3000)

And I found that the problem was still present.  My config.h is attached.

I diagnosed the cause.  It was due to the initial ESC byte of an arrow 
key sequence (\eOC for right or \eOD for left arrow) being interpreted 
as a press of the escape key, and the remainder as commands which clear 
to the end of the line.

Normally readline will wait an unlimited amount of time for additional 
input, and will only react once it is able to resolve this kind of 
ambiguity between different bound input sequences.  For this special 
case of the escape key however, there is a special case in the code to 
give instant response, if there is nothing in the input queue after the 
escape byte.  Otherwise the vi-mode user would not see a reaction to 
hitting the escape key until they gave additional input, and that would 
not be nice.
Unfortunately, depending on your exact configuration, it may be 
possible, and may even be reasonably probable for the end of the input 
queue to be in the middle of an input sequence for a single key, when it 
is read.  In this situation the arrow key can be misinterpreted as 
described above.
I believe the chance of the problem may increase when bash is on the 
other side of a slow network link.

The editor vi uses a timeout system to deal with this, but that would 
require significant changes to the code, and even then it's not a 
perfect solution.

To eliminate the ambiguity, I told my terminal to generate a longer 
sequence (\eO[) when I hit escape.  (For anyone interested, my terminal 
is mintty, I believe emulating an xterm, and I used this command: echo 
-e '\e[?7727h')
I set up appropriate readline bindings.

Then I added a variable "vi-escape-hack" to readline, which I turned off 
in my .inputrc to remove the special handling of escape.
The option is on by default, so default behaviour is unchanged.

After that I was able to use the arrow keys to navigate the line without 
the problem occurring again.
Note though that since I can't unbind "\e" in .inputrc, it might still 
be possible to accidentally trigger that via a longer sequence that 
hasn't been bound to anything, but I haven't seen this.

I was hoping to get the vi-escape-hack option added to the official 
readline code.  The patch is very simple.  Just a few lines.  It is 
based on readline-6.2.  I named the variable as best I could, but I 
wouldn't mind if the name were changed.

vi-escape-hack.patch
Description: Text document

config.h
Description: Text document