bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max

[Eli Zaretskii]

> Date: Thu, 21 Sep 2023 03:57:43 +0300
> Cc: 66020@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> That leaves the question of what new value to use. 409600 is optimal for 
> a large-output process but seems too much as default anyway (even if I 
> have very little experimental proof for that hesitance: any help with 
> that would be very welcome).

How does the throughput depend on this value?  If the dependence curve
plateaus at some lower value, we could use that lower value as a
"good-enough" default.

> I did some more experimenting, though. At a superficial glance, 
> allocating the 'chars' buffer at the beginning of read_process_output is 
> problematic because we could instead reuse a buffer for the whole 
> duration of the process. I tried that (adding a new field to 
> Lisp_Process and setting it in make_process), although I had to use a 
> value produced by make_uninit_string: apparently simply storing a char* 
> field inside a managed structure creates problems for the GC and early 
> segfaults. Anyway, the result was slightly _slower_ than the status quo.
> 
> So I read what 'alloca' does, and it looks hard to beat. But it's only 
> used (as you of course know) when the value is <= MAX_ALLOCA, which is 
> currently 16384. Perhaps an optimal default value shouldn't exceed this, 
> even if it's hard to create a benchmark that shows a difference. With 
> read-process-output-max set to 16384, my original benchmark gets about 
> halfway to the optimal number.

Which I think means we should stop worrying about the overhead of
malloc for this purpose, as it is fast enough, at least on GNU/Linux.

> And I think we should make the process "remember" the value at its 
> creation either way (something touched on in bug#38561): in bug#55737 we 
> added an fcntl call to make the larger values take effect. But this call 
> is in create_process: so any subsequent increase to a large value of 
> this var won't have effect.

Why would the variable change after create_process?  I'm afraid I
don't understand what issue you are trying to deal with here.