bug#66117: 30.0.50; `find-buffer-visiting' is slow when opening large number of buffers

[Dmitry Gutov]

On 22/09/2023 14:03, Ihor Radchenko wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
>
> > > Would it be possible to implement some kind of caching mechanism to be
> > > used by `find-buffer-visiting'?
> >
> > I'm guessing you Cc'd me because of an existing comment inside xref.el?
>
> Actually, mostly because I though that this discussion might be of
> interest for you.

All right, thank you for that.

> > As you can see I decided not to use this function there, but even
> > get-file-buffer wasn't as fast as I would've wanted, so there's a
> > quick-and-dirty caching solution for sequential lookups (which assumes
> > that the same file would be looked up multiple times in a row).
>
> For me, the situation is different - a new not-yet-open file is looked
> up multiple times in a row, which is a bit more tricky.

This could be handled with a locally held dictionary as well, but...

> > Putting aside the perspective of maintaining a cache in the core
> > (hopefully this will be discussed later in this feature request), any
> > chance you could initialize that cache yourself first (by iterating
> > through buffers and building file-truename -> buffer map), before you
> > are doing those many lookups? Somewhere in org-agenda-list, perhaps.
> >
> > That should change the complexity from N*M to N+M, more or less.
>
> Sure. That's my plan to address older Emacs versions. But such iteration
> will involve copy-pasting a big part of `find-buffer-visiting' source
> code, which is not something I want to maintain in a long term (what if
> `find-buffer-visiting' changes in future?).
>
> Moreover, `find-buffer-visiting' is called by `find-file-noselect', and
> I simply cannot make use of cache there without modifying the code
> upstream; or writing my own version of `find-file-noselect' - bad idea
> maintenance-wise.

...if most of said calls are done through find-file-noselect, I suppose 
that solution is a no-go.

> I think that the best way that will benefit more than Org mode is
> arranging internal cache that will link buffer-file-name,
> buffer-file-truename, and buffer-file-number with buffers; and maintain
> the correctness of the cache if buffer-file-name changes for any reason.

I think that is doable.

It probably won't regress the performance of any particular scenario 
either, but we should benchmark opening a bunch of files this way anyway 
(might actually get faster, due to find-buffer-visiting calls).