bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Dmitry Gutov]

On 11/09/2023 14:57, Eli Zaretskii wrote:
> > Date: Mon, 11 Sep 2023 03:02:55 +0300
> > Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
> >   64735@debbugs.gnu.org
> > From: Dmitry Gutov <dmitry@gutov.dev>
> >
> > > You could record its value in a local variable at the entry to
> > > garbage_collect, and the expose that value to Lisp.
> >
> > That also doesn't seem to give much, given that the condition for
> > entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait
> > until it's down to 0, then garbage-collect.
>
> No, we don't wait until it's zero, we perform GC on the first
> opportunity that we _notice_ that it crossed zero.  So examining how
> negative is the value of consing_until_gc when GC is actually
> performed could tell us whether we checked the threshold with high
> enough frequency, and comparing these values between different runs
> could tell us whether the shorter time spend in GC means really less
> garbage or less frequent checks for the need to GC.

Good point, I'm attaching the same outputs with "last value of 
consing_until_gc" added to every line.

There are some pretty low values in the "read-process-output-max 409600" 
part of the experiment, which probably means runtime staying in C 
accumulating the output into the (now larger) buffer? Not sure.

> > What's in there? First of all, for find-directory-files-recursively-3,
> > there are 0 garbage collections between the beginning of the function
> > and when we start parsing the output (no GCs while the process is
> > writing to the buffer synchronously). I guess inserting output in a
> > buffer doesn't increase consing, so there's nothing to GC?
>
> No, we just don't count increasing size of buffer text in the "consing
> since GC" counter.  Basically, buffer text is never "garbage", except
> when a buffer is killed.

That makes sense. Perhaps it hints at a faster design for calling 
process asynchronously as well (more on another experiment later).

> > Next: for find-directory-files-recursively-2, the process only finishes
> > at the end, when all GC cycles are done for. I suppose that also means
> > we block the process's output while Lisp is running, and also that
> > whatever GC events occur might coincide with the chunks of output coming
> > from the process, and however many of them turn out to be in total.
>
> We don't block the process when GC runs.  We do stop reading from the
> process, so if and when the pipe fills, the OS will block the process.

Right. But the effect is almost the same, including the potential 
side-effect that (IIUC) when a process is waiting like that, it's just 
suspended and not rushing ahead using the CPU/disk/etc resources to the 
max. That's an orthogonal train of thought, sorry.

gcs2b.txt
Description: Text document

gcs3b.txt
Description: Text document