Re: Performance issue in project.el

[Ergus]

On Sun, Jun 27, 2021 at 11:21:23PM +0200, Ergus wrote:
> Hi:
>
> Using tramp I tried to use project.el with a command like
> project-switch-to-buffer and it took like 10 minutes to complete.
>
> I ran a profiler and I found that most of the time was taken by:
>
> project-current called in a loop for all the opened buffers it calls
> project--find-in-directory that calls
> project-find-functions and there is going all the time.
>
> I had global-tags-try-project-root (a function from an external package
> tat call gnu global program... something like gtags). And I optimized
> that function to use a cache and avoid calling the program for every
> buffer creating a cache for it.
>
> Something like:
>
> ```
> (defvar global-tags--dbpath-cache (make-hash-table))
>
> (cl-defun global-tags--get-dbpath (&optional (dir default-directory))
>  "Filepath for database from DIR."
>  (pcase (gethash dir global-tags--dbpath-cache)
>    ('empty nil)
>    ('nil
>     (puthash dir 'empty global-tags--dbpath-cache)
>     (when-let*
>       === BLA ===
>       (when (file-exists-p dbpath)
>          (puthash dir dbpath global-tags--dbpath-cache))))
>    (code code)))
> ````
>
> Now the time is half than before but still very slow to use the command
> (around 3-5 minutes to complete) and running again the profiler I get this:
>
>        5637  89% - command-execute
>        5549  88%  - byte-code
>        5549  88%   - project--read-project-buffer
>        5549  88%    - let*
>        5336  85%     - read-buffer
>        5323  84%      - ivy-completing-read
>        5323  84%       - ivy-read
>        4941  78%        - ivy--reset-state
>        4941  78%         - ivy--buffer-list
>        4941  78%          - internal-complete-buffer
>        4941  78%           - #<lambda -0x1a357caf01243d61>
>        4941  78%            - and
>        4941  78%             - equal
>        4941  78%              - save-current-buffer
>        4941  78%               - project-current
>        4941  78%                - project--find-in-directory
>        4548  72%                 - project-try-vc
>        4537  72%                  - vc-responsible-backend
>        4478  71%                   - #<compiled 0xd3f2e32af0966f7>
>        4478  71%                    - vc-call-backend
>        4478  71%                     - apply
>        1470  23%                      + vc-svn-responsible-p
>        1142  18%                      + vc-bzr-responsible-p
>         970  15%                      + vc-hg-responsible-p
>         390   6%                      + vc-git-responsible-p
>         156   2%                      + vc-cvs-responsible-p
>         126   2%                      + vc-rcs-responsible-p
>         108   1%                      + vc-sccs-responsible-p
>          98   1%                      + vc-src-responsible-p
>          57   0%                   + tramp-file-name-handler
>          11   0%                  + vc-file-getprop
>         393   6%                 + global-tags-try-project-root
>         375   5%        + read-from-minibuffer
>          13   0%      + if
>         213   3%     + project-current
>          88   1%  + funcall-interactively
>         572   9% + ...
>          51   0% + timer-event-handler
>           8   0% + redisplay_internal (C function)
>
>
> As you can see most of the time is still taken by project-current and I
> can't really understand why:
>
> 1) Are so many samples 4548 seems a very high number for only 25 opened
> buffers.
>
> 2) why project-try-vc still takes so much...? Specially for unfrequent
> vc systems in our days like svn or bzr that I am not using.
>
> 3) If vc-handled-backends has these value by default: '(RCS CVS SVN SCCS
> SRC Bzr Git Hg Mtn) why is there some time taken by vc-hg-responsible-p
> that is latter than git in the list?
Just replying to myself about 3)

After reading the function vc-responsible-backend I can see that it uses
a mapcar to test the responsible; so all of them are always tested in
every single call every time.

As a workaround I removed all the uninteresting handlers from
vc-handled-backends and I get better times now, but IMHO it is still
very inefficient almost a minute for project-switch-to-buffer is
excessive.

VCS changing is not something that happens very often to require a check
of all the backends everytime, several times for every buffer in many
project.el functions right?

Should I open an issue for this?

Maybe it is possible to add a sort of optimization like the one
mentioned above to vc-responsible-backend?

Probably I am wrong, but: Do we really need to check all this everytime
we call project-current?  I mean, a file or default-dir changes project
so often that we need to check with all the backends constantly?

Either in the local system the performance penalty seems to be
significant.

> In any case:
>
> Maybe (I think I mentioned this before) `project.el` needs a sort of
> cache to speedup some functions like `project-current` that are called
> very frequently.
>
> Best,
> Ergus