Re: Increase FD_SETSIZE in w32.h to allow more than 32 (actually 29) subprocesses

[Eli Zaretskii]

> From: "Yue Yi" <include_yy@qq.com>
> Cc: "emacs-devel" <emacs-devel@gnu.org>
> Date: Tue, 28 Jan 2025 15:21:20 +0800
> 
> > > Based on the implementation of Emacs's inter-process communication in
> > > Windows, we need to wait for both the subprocess itself and a
> > > pipe. Therefore, this implementation can wait for a maximum of
> > > 4096/2=2048 subprocesses. However, to remain consistent with the number
> > > of waits supported by `select', I have limited the maximum FD_SIZE to
> > > 2048. As a result, the maximum number of subprocesses is reduced to
> > > 1024. 2048 comes from (MAXIMUM_WAIT_OBJECTS * MAXIMUM_WAIT_OBJECTS / 2)
> > 
> > I'm not sure I understand the "to remain consistent with the number of
> > waits supported by `select'" part.  Do you mean 'select' as we use it
> > in process.c?  Or do you mean something else?  What is that
> > limitation?
> > 
> > Of course, all of the above should be in the comments to explain the
> > code.
> 
> Thank you for your feedback, and I apologize for the lack of clarity in
> my previous explanation. Let me clarify the rationale behind the
> FD_SETSIZE value.
> 
> On Linux, FD_SETSIZE is typically 1024 by default, and it supports
> managing at most 1024 subprocesses via 'pty method. By setting
> FD_SETSIZE = 2048 in my code, the effective maximum subprocess capacity
> becomes 2048 / 2 = 1024 on Windows, aligning with Linux's default scale.
> 
> In my original thoughts, This design aims to approximate cross-platform
> behavior consistency. This may be unnecessary. I could set it to the
> maximum number of objects my implementation can wait for: 64 * 63.

If the idea is not to go beyond what GNU/Linux users have, let's just
use 2048, and add a comment explaining it as provision for 1024
subprocess as on GNU/Linux.  There's no need to "encipher" 2048 as
64 * 64 / 2.

> After some simple investigation, I believe the overhead of creating
> new threads for each call to sys_select might be unacceptable.
> 
> The link https://stackoverflow.com/a/18275460 provides a good piece of
> code for testing the time taken for thread creation. The test results
> given in the link show that the time from thread creation to actual
> execution ranges from tens to hundreds of microseconds. I slightly
> modified the code and tested the time taken for creating and executing
> 64 threads. Below are my test results:
> 
> total:  34.581500 ms, average:   0.540336 ms
> max  :   1.429200 ms, min    :   0.039000 ms
> 
> I can modify `sys_select' to observe its call time, but I opted for a
> simpler approach: using `accept-process-output':
> 
> (benchmark-run 1000000 (accept-process-output))
> => (1.4010608 4 0.2107215)
> 
> The above results indicate that the call time of `sys_select' should be
> less than 2 microseconds. However, when I applied my patch and created
> more than 32 child processes, this value ballooned to an astonishing 160
> microseconds.
> 
> I'm planning to continue my thread pool implementation. WDYT?

Can you describe in short how this will work from the thread
management POV?  E.g., when (under what circumstances) will the
threads be created, and will they ever exit or be killed?  Our needs
are quite unique, I think, so perhaps it would make sense to have some
special thread management policies (as opposed to what standard thread
pool implementations provide)?  Not that I'm an expert on thread
pools, so maybe what I say here makes little sense.

Thanks.