bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Eli Zaretskii]

> Date: Mon, 11 Sep 2023 03:02:55 +0300
> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
>  64735@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > You could record its value in a local variable at the entry to
> > garbage_collect, and the expose that value to Lisp.
> 
> That also doesn't seem to give much, given that the condition for 
> entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait 
> until it's down to 0, then garbage-collect.

No, we don't wait until it's zero, we perform GC on the first
opportunity that we _notice_ that it crossed zero.  So examining how
negative is the value of consing_until_gc when GC is actually
performed could tell us whether we checked the threshold with high
enough frequency, and comparing these values between different runs
could tell us whether the shorter time spend in GC means really less
garbage or less frequent checks for the need to GC.

> What's in there? First of all, for find-directory-files-recursively-3, 
> there are 0 garbage collections between the beginning of the function 
> and when we start parsing the output (no GCs while the process is 
> writing to the buffer synchronously). I guess inserting output in a 
> buffer doesn't increase consing, so there's nothing to GC?

No, we just don't count increasing size of buffer text in the "consing
since GC" counter.  Basically, buffer text is never "garbage", except
when a buffer is killed.

> Next: for find-directory-files-recursively-2, the process only finishes 
> at the end, when all GC cycles are done for. I suppose that also means 
> we block the process's output while Lisp is running, and also that 
> whatever GC events occur might coincide with the chunks of output coming 
> from the process, and however many of them turn out to be in total.

We don't block the process when GC runs.  We do stop reading from the
process, so if and when the pipe fills, the OS will block the process.

> So there is also a second recording for 
> find-directory-files-recursively-2 with read-process-output-max=409600. 
> It does improve the performance significantly (and reduce the number of 
> GC pauses). I guess what I'm still not clear on, is whether the number 
> of GC pauses is fewer because of less consing (the only column that 
> looks significantly different is the 3rd: VECTOR-CELLS), or because the 
> process finishes faster due to larger buffers, which itself causes fewer 
> calls to maybe_gc.

I think the latter.