ftell, ftello: Fix recognition of pipes on native Windows

[Bruno Haible]

ftell() and ftello() must fail when the stream refers to a pipe, says POSIX
<https://pubs.opengroup.org/onlinepubs/9699919799/functions/ftell.html>.
Likewise for fgetpos()
<https://pubs.opengroup.org/onlinepubs/9699919799/functions/fgetpos.html>.

Gnulib checks this for ftell() and ftello(). These two tests succeed in
an environment with mingw 5 on Cygwin 2.9.0 + Windows 21H2, but fail in
an environment with mingw 10 on Cygwin 3.4.6 + Windows 22H2.

More precisely, what fails are the executions of
  $ echo hi | ./test-ftell
and
  $ echo hi | ./test-ftello

The attached program foo.c tests the result of ftell(), ftello(),
fgetpos(), _lseeki64(), GetFileType() on stdin.

I compiled this program on the old environment, and it exhibits a
different behaviour in the new than in the old environment.

When compiled with mingw, the binary depends on kernel32.dll and msvcrt.dll;
when compiled with MSVC, the binary depends on kernel32.dll only. (Verified
with "dumpbin /imports".)
So, when this program behaves differently in the new environment, it
must be due to environment changes other than the msvcrt.

More precisely:

In Cygwin 2.9.0 (old environment):
$ echo hi | ./foo-old-mingw.exe
ftell() -> -1
ftell0() -> -1
fgetpos -> fail
_lseeki64 -> -1
SetFilePointerEx -> fail
GetFileType -> FILE_TYPE_PIPE
$ echo hi | ./foo-old-msvc.exe
ftell() -> -1
_lseeki64 -> -1
SetFilePointerEx -> fail
GetFileType -> FILE_TYPE_PIPE

In Cygwin 3.4.6 (new environment)
and in cmd.exe (both in the old and new environments):
$ echo hi | ./foo-old-mingw.exe
ftell() -> 0
ftell0() -> 0
fgetpos -> success
_lseeki64 -> 0
SetFilePointerEx -> 0
GetFileType -> FILE_TYPE_PIPE
$ echo hi | ./foo-old-msvc.exe
ftell() -> 0
_lseeki64 -> 0
SetFilePointerEx -> 0
GetFileType -> FILE_TYPE_PIPE

So, although the handle is of type FILE_TYPE_PIPE, SetFilePointerEx does not
fail on it. This contradicts
https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setfilepointerex
  "You cannot use the SetFilePointerEx function with a handle to a nonseeking
   device such as a pipe or a communications device. To determine the file
   type for hFile, use the GetFileType function."

It's not clear to me whether the change in behaviour is due to
changes in kernel.dll (between Windows 21H2 and 22H2) or due to the
info that the parent process (bash) passes to the second process in the
pipe (maybe different between Cygwin 2.9.0 and 3.4.6?). But this does
not matter. What matters is that relying on SetFilePointerEx to fail
is unreliable. And likewise, relying on _lseeki64 to fail is unreliable.

Gnulib's module lseek() already contains the necessary workaround. All
that remains to be done it to override ftell() and ftello(), so that they
get the necessary workaround as well.

Note: When these programs are applied the terminal as stdin, there is a
difference in behaviour as well:

In Cygwin 2.9.0:
$ ./foo-old-msvc.exe
ftell() -> 0
_lseeki64 -> 0
SetFilePointerEx -> 0
GetFileType -> FILE_TYPE_PIPE

In Cygwin 3.4.6:
$ ./foo-old-msvc.exe
ftell() -> 0
_lseeki64 -> 0
SetFilePointerEx -> 0
GetFileType -> FILE_TYPE_CHAR

GetFileType() returns a different value. Fortunately, this does not make
a difference for Gnulib.


2023-04-24  Bruno Haible  <bruno@clisp.org>

        ftell, ftello: Fix recognition of pipes on native Windows.
        * m4/lseek.m4 (gl_FUNC_LSEEK): Update comment.
        * m4/ftello.m4 (gl_FUNC_FTELLO): On native Windows, set REPLACE_FTELLO=1
        always.
        * doc/posix-functions/ftello.texi: Mention the behaviour on pipes.
        * doc/posix-functions/ftell.texi: Likewise.
        * doc/posix-functions/fgetpos.texi: Likewise.

diff --git a/doc/posix-functions/fgetpos.texi b/doc/posix-functions/fgetpos.texi
index 89e3acd58f..4a0051ea23 100644
--- a/doc/posix-functions/fgetpos.texi
+++ b/doc/posix-functions/fgetpos.texi
@@ -19,4 +19,7 @@
 @code{fflush}, @code{ftell}, @code{ftello}, @code{fgetpos} behave incorrectly
 on input streams that are opened in @code{O_TEXT} mode and whose contents
 contains Unix line terminators (LF), on some platforms: mingw, MSVC 14.
+@item
+This function mistakenly succeeds on pipes on some platforms:
+mingw 10.
 @end itemize
diff --git a/doc/posix-functions/ftell.texi b/doc/posix-functions/ftell.texi
index 5a15dee8ea..5d79f5576b 100644
--- a/doc/posix-functions/ftell.texi
+++ b/doc/posix-functions/ftell.texi
@@ -20,6 +20,9 @@
 buffer, changes the stream's file position, and returns the wrong position on
 some platforms:
 macOS 10.15 and newer.
+@item
+This function mistakenly succeeds on pipes on some platforms:
+mingw 10.
 @end itemize
 
 Portability problems not fixed by Gnulib:
diff --git a/doc/posix-functions/ftello.texi b/doc/posix-functions/ftello.texi
index ded1df0c11..4e9427015c 100644
--- a/doc/posix-functions/ftello.texi
+++ b/doc/posix-functions/ftello.texi
@@ -27,6 +27,9 @@
 @item
 This function fails on seekable stdin, stdout, and stderr: cygwin <= 1.5.24.
 @item
+This function mistakenly succeeds on pipes on some platforms:
+mingw 10.
+@item
 On platforms where @code{off_t} is a 32-bit type, @code{ftello} does not work
 correctly with files 2 GiB and larger.  @xref{Large File Support}.
 @end itemize
diff --git a/m4/ftello.m4 b/m4/ftello.m4
index 4901b16835..e13fcd93d2 100644
--- a/m4/ftello.m4
+++ b/m4/ftello.m4
@@ -1,4 +1,4 @@
-# ftello.m4 serial 14
+# ftello.m4 serial 15
 dnl Copyright (C) 2007-2023 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,13 +37,24 @@ AC_DEFUN([gl_FUNC_FTELLO]
     if test $gl_cv_var_stdin_large_offset = no; then
       REPLACE_FTELLO=1
     fi
+    AC_REQUIRE([AC_CANONICAL_HOST])
+    if test $REPLACE_FTELLO = 0; then
+      dnl On native Windows, in some circumstances, ftell(), ftello(),
+      dnl fgetpos(), lseek(), _lseeki64() all succeed on devices of type
+      dnl FILE_TYPE_PIPE. However, to match POSIX behaviour, we want
+      dnl ftell(), ftello(), fgetpos(), lseek() to fail when the argument fd
+      dnl designates a pipe. See also
+      dnl 
https://github.com/python/cpython/issues/78961#issuecomment-1093800325
+      case "$host_os" in
+        mingw*) REPLACE_FTELLO=1 ;;
+      esac
+    fi
     if test $REPLACE_FTELLO = 0; then
       dnl Detect bug on Solaris.
       dnl ftell and ftello produce incorrect results after putc that followed a
       dnl getc call that reached EOF on Solaris. This is because the _IOREAD
       dnl flag does not get cleared in this case, even though _IOWRT gets set,
       dnl and ftell and ftello look whether the _IOREAD flag is set.
-      AC_REQUIRE([AC_CANONICAL_HOST])
       AC_CACHE_CHECK([whether ftello works],
         [gl_cv_func_ftello_works],
         [
diff --git a/m4/lseek.m4 b/m4/lseek.m4
index 6e1ab6ffaa..0da458804f 100644
--- a/m4/lseek.m4
+++ b/m4/lseek.m4
@@ -1,4 +1,4 @@
-# lseek.m4 serial 13
+# lseek.m4 serial 14
 dnl Copyright (C) 2007, 2009-2023 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -17,9 +17,11 @@ AC_DEFUN([gl_FUNC_LSEEK]
          dnl Native Windows.
          dnl The result of lseek (fd, (off_t)0, SEEK_CUR) or
          dnl SetFilePointer(handle, 0, NULL, FILE_CURRENT)
-         dnl for a pipe depends on the environment: In a Cygwin 1.5
-         dnl environment it succeeds (wrong); in a Cygwin 1.7 environment
-         dnl it fails with a wrong errno value.
+         dnl for a pipe depends on the environment:
+         dnl In a Cygwin 1.5 environment it succeeds (wrong);
+         dnl in a Cygwin 1.7 environment it fails with a wrong errno value;
+         dnl in a Cygwin 2.9.0 environment it fails correctly;
+         dnl in a Cygwin 3.4.6 environment it succeeds again (wrong).
          gl_cv_func_lseek_pipe=no
          ;;
        *)

foo.c
Description: Text Data