[Bug-readline] readline patch from gdb

[Tom Tromey]

Hi.  I recently looked at updating gdb to readline 7.0 patch 5
(from readline 6.2 patch 1).

You can see a full report of the gdb side here:

https://sourceware.org/ml/gdb-patches/2018-09/msg00930.html

One specific thing that came out of this is that the win32_isatty
code still seems to need a change.  I've appended the patch.

My current merge tree is here:

https://github.com/tromey/gdb-readline/tree/gdb-readline

The other changes between readline and gdb seem to be changes for gdb,
some of which can probably be removed.  Those need investigation.

In my manual patch diffing I found some changes to #ifs but those didn't
survive the merge.

This patch was written by Eli Zaretskii:

commit 7f3c5ec870943f7f32c946ff9459dfd04fcb8e07
Author: Eli Zaretskii <address@hidden>
Date:   Sat Sep 17 11:50:37 2016 +0300

    Improve MinGW support in Readline
    
    These changes were already accepted upstream in Readline,
    but GDB did not yet import a newer Readline version.
    
    readline/Changelog.gdb:
    
            * util.c: Include rlshell.h.
            (_rl_tropen) [_WIN32 && !__CYGWIN__]: Open the trace file in the
            user's temporary directory.
            * tcap.h [HAVE_NCURSES_TERMCAP_H]: Include ncurses/termcap.h.
            * input.c (w32_isatty) [_WIN32 && !__CYGWIN__]: New function, to
            replace isatty that is not reliable enough on MS-Windows.
            (isatty) [_WIN32 && !__CYGWIN__]: Redirect to w32_isatty.
            (rl_getc): Call _getch, not getch, which could be an ncurses
            function when linked with ncurses, in which case getch will return
            EOF for any keystroke, because there's no curses window.
            * tilde.c (tilde_expand_word) [_WIN32]:
            * histfile.c (history_filename) [_WIN32]: Windows-specific
            environment variable to replace HOME if that is undefined.
            * funmap.c (default_funmap): Compile rl_paste_from_clipboard on
            all Windows platforms, not just Cygwin.
            * readline.h (rl_paste_from_clipboard): Include declaration for
            all Windows platforms.
            * display.c (insert_some_chars, delete_chars): Don't use the
            MinGW-specific code if linked with ncurses.
            * configure.in:
            * config.h.in: Support ncurses/termcap.h.  The configure script
            was updated accordingly.
            * complete.c [_WIN32 && !__CYGWIN__]: Initialize
            _rl_completion_case_fold to 1.
            (printable_part, rl_filename_completion_function)
            [_WIN32 && !__CYGWIN__]: Handle the drive letter.

Tom

diff --git a/input.c b/input.c
index 24126ea..077d53c 100644
--- a/input.c
+++ b/input.c
@@ -95,20 +95,35 @@ static int ibuffer_space PARAMS((void));
 static int rl_get_char PARAMS((int *));
 static int rl_gather_tyi PARAMS((void));
 
-/* Windows isatty returns true for every character device, including the null
-   device, so we need to perform additional checks. */
 #if defined (_WIN32) && !defined (__CYGWIN__)
+
+/* 'isatty' in the Windows runtime returns non-zero for every
+   character device, including the null device.  Repair that.  */
 #include <io.h>
+#include <conio.h>
 #define WIN32_LEAN_AND_MEAN 1
 #include <windows.h>
 
-int
-win32_isatty (int fd)
+int w32_isatty (int fd)
 {
-  return (_isatty (fd) ? ((((long) (HANDLE) _get_osfhandle (fd)) & 3) == 3) : 
0);
+  if (_isatty(fd))
+    {
+      HANDLE h = (HANDLE) _get_osfhandle (fd);
+      DWORD ignored;
+
+      if (h == INVALID_HANDLE_VALUE)
+       {
+         errno = EBADF;
+         return 0;
+       }
+      if (GetConsoleMode (h, &ignored) != 0)
+       return 1;
+    }
+  errno = ENOTTY;
+  return 0;
 }
 
-#define isatty(x)      win32_isatty(x)
+#define isatty(x)  w32_isatty(x)
 #endif
 
 /* **************************************************************** */