Re: Increase FD_SETSIZE in w32.h to allow more than 32 (actually 29) subprocesses

[Eli Zaretskii]

> From: "Yue Yi" <include_yy@qq.com>
> Cc: "emacs-devel" <emacs-devel@gnu.org>
> Date: Thu, 30 Jan 2025 08:24:22 +0800
> 
> > Can you describe in short how this will work from the thread
> > management POV?  E.g., when (under what circumstances) will the
> > threads be created, and will they ever exit or be killed?  Our needs
> > are quite unique, I think, so perhaps it would make sense to have some
> > special thread management policies (as opposed to what standard thread
> > pool implementations provide)?  Not that I'm an expert on thread
> > pools, so maybe what I say here makes little sense.
> 
> Sure.
> 
> A thread pool is a collection of pre-initialized threads that are ready
> to perform tasks. Instead of creating and destroying threads for each
> task, a thread pool reuses a fixed number of threads to handle multiple
> tasks. This improves performance and resource efficiency by reducing the
> overhead of thread creation and destruction.
> 
> For Emacs, my current implementation involves creating an array of
> structures with a length of 32, where each element contains contextual
> information for a thread. Threads are only created when necessary, such
> as when the number of child processes exceeds 32 (or more) for the first
> time. Once created, the threads do not exit; instead, they enter an
> infinite loop, waiting for an event to trigger and begin their
> WaitForMultipleObjects tasks.

OK, thanks.  It's possible that if a long time passes without using
more than M threads from the pool, we could cause N - M threads to
exit (where N is the total number of threads in the pool), to reduce
the amount of resources used by the Emacs process.  But this is an
optimization, and can be considered later.

> In this implementation, when there are a large number of child processes
> (around 1000), the maximum call time for accept-process-output does not
> exceed 100 microseconds. However, there might still be room for
> improvement.

AFAIR, you said that starting new threads each time meant the timing
was 160 microseconds (vs just 2 usec without threads)?  If so, the
improvement due to using thread pool is a bit disappointing, no?  If
the time cannot be reduced to, say, 25 usec (which is already 10 times
the current performance), then maybe the additional complexities of
using a thread pool are not justified, and we should live with the
slower subprocess responses when there's more than 30 of them?

> By the way, can I use _WIN32_WINNT to check the current Windows version
> and make the most of some system features, such as the thread pool? I
> will compare the performance of the system thread pool with my own
> implementation to decide whether to use them.

Yes, we could decide that Emacs running on older Windows versions will
not be able to use this new feature, or some of its advanced aspects,
if the newer Windows versions provide some functionality that can
significantly improve run-time performance.  However, _WIN32_WINNT is
not the right tool for these decisions, because it's a compile-time
decision, whereas Emacs binaries for Windows are in many cases
produced on a Windows system different from the one where Emacs is
eventually run.  So it is quite possible that Emacs is built on
Windows 11 and then run on Windows 8.1 or XP or even 9X.

This is why we usually prefer to have _WIN32_WINNT set to the default
value of the MinGW environment, which is presumably the lowest Windows
version supported by MinGW used to compile Emacs.  If we need
compile-time or run-time features that are only available on later
Windows versions, we either

  . include the necessary portions of the headers in our sources (to
    work around problems with macros or functions or data types
    defined only for higher values of _WIN32_WINNT), or
  . attempt to load relevant DLLs at run time and call the functions
    from them via function pointers, or
  . provide less performant alternatives for older Windows versions by
    calling alternative APIs, conditioned on either the Windows
    version detected at run time or by the fact that loading a DLL
    and/or finding a function inside that DLL and/or calling that
    function succeeded or failed

You will find many examples of using these techniques in w32*.c source
files, and we can discuss what to do in specific situations you
encounter as part of working on this feature.