bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Eli Zaretskii]

> Date: Wed, 13 Sep 2023 23:38:29 +0300
> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
>  64735@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > I think these questions are slightly premature.  We should first have
> > the implementation of that filter, and then look for candidates that
> > could benefit from it.
> 
> The implementation in that patch looks almost complete to me, unless you 
> have any further comments.

Fine, then please post a complete patch with all the bells and
whistles, and let's have it reviewed more widely.  (I suggest a new
bug report, as this one is already prohibitively long to follow,
includes unrelated issues, and I fear some people will ignore patches
posted to it).  I think there are a few subtleties we still need to
figure out.

> The main difference would be the change in 
> the dispatch comparison from
> 
>    if (p->filter == Qinternal_default_process_filter)
> 
> to
> 
>    if (p->filter == Qbuffer)

Btw, both of the above are mistakes: you cannot compare Lisp objects
as if they were simple values.  You must use EQ.

>     This function gives PROCESS the filter function FILTER.  If FILTER
>       is ‘nil’, it gives the process the default filter, which inserts
>       the process output into the process buffer.  If FILTER is ‘t’,
>       Emacs stops accepting output from the process, unless it’s a
>       network server process that listens for incoming connections.
> 
> What can we add?
> 
>    If FILTER is ‘buffer’, it works like the default one, only a bit faster.
> 
> ?

  If FILTER is the symbol ‘buffer’, it works like the default filter,
  but makes some shortcuts to be faster: it doesn't adjust markers and
  the process mark (something else?).

Of course, the real text will depend on what the final patch will look
like: I'm not yet sure I understand which parts of
internal-default-process-filter you want to keep in this alternative
filter.  (If you intend to keep all of them, it might be better to
replace internal-default-process-filter completely, perhaps first with
some variable exposed to Lisp which we could use to see if the new one
causes issues.)

> > My tendency is to change only callers which
> > are in many cases expected to get a lot of stuff from a subprocess, so
> > shell buffers are probably out.  But we could discuss that later.
> 
> When I'm thinking of start-file-process-shell-command, I have in mind 
> project--files-in-directory, which currently uses 
> process-file-shell-command. Though I suppose most cases would be more 
> easily converted to use make-process (like xref-matches-in-files uses 
> process-file for launching a shell pipeline already).
> 
> I was also thinking about Flymake backends because those work in the 
> background. The outputs are usually small, but can easily grow in rare 
> cases, without particular limit. Flymake also runs in the background, 
> meaning whatever extra work it has to do (or especially GC pressure), 
> affects the delays when editing.

I think we will have to address these on a case by case basis.  The
issues and aspects are not trivial and sometimes subtle.  We might
even introduce knobs to allow different pipe sizes if there's no
one-fits-all value for a specific function using these primitives.

> >> I'm not sure what negatives to test for, though. Raising the limit 10x
> >> is unlikely to lead to an OOM, but I guess some processes could grow
> >> higher latency?..
> > 
> > With a large buffer and small subprocess output we will ask the OS for
> > a large memory increment for no good reason.  Then the following GC
> > will want to compact the gap, which means it will be slower.
> 
> I wonder what scenario that might become apparent in. Launching many 
> small processes at once? Can't think of a realistic test case.

One process suffices.  The effect might not be significant, but
slowdowns due to new features are generally considered regressions.