bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time

[Eli Zaretskii]

> Date: Fri, 24 Sep 2021 09:00:13 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Dmitry Gutov <dgutov@yandex.ru>, 50733@debbugs.gnu.org, mardani29@yahoo.es
> 
> > Btw, I don't understand why we focus on general-purpose text-searching 
> > tools for these features.  Why not focus on packages like ID Utils 
> > instead, they are so much faster.  Daniel, could you time the same 
> > search in that large tree when xref-search-program is 'gid'?  (You'd 
> > need to run 'mkid' first, to create the ID database, but that is 
> > one-time, and is very fast.)  As I told many times, I think this is the 
> > future: program language sensitive tools that use a precomputed DB.
> 
> id-tools are useful, but they do not provide the same functionality as 
> grep/ripgrep.  On Emacs' trunk, mkid takes a few seconds to create a 12 MB 
> database, and gid Eli returns only 4 occurrences.
> 
> This can be improved by using mkid --default-lang=text, but then mkid 
> takes no less than 30 minutes (!) to create a 205 MB database, and gid Eli 
> now returns 25 occurrences in ~1 second.
> 
> In comparison, rg Eli returns 6474 occurrences in < 0.1 seconds.

We are talking about project.el, so the focus should be on looking for
strings that are meaningful in the programming-language context,
because that is what we need when programming.  Not looking for
arbitrary strings.  If someone wants to look for arbitrary strings,
they want "M-x grep" (with or without ripgrep), not
project-find-regexp.  The meaning of "word" or "symbol" is different
in different PLs; a tool like Grep can only approximate that, whereas
ID Utils uses the correct definition for each language.

> Moreover, incremental updates are not implemented in mkid.

It doesn't matter, as it's fast enough, and putting it on some kind of
cron job is more than enough to allow forgetting that this step
exists.

TANSTAAFL, so please let's not insist on looking for one.  We won't
find it.  Every alternative will let you pay something, so the issue
at hand is to find the alternative that is the best on balance, for
this particular feature, not one which is free of any price.

That said, you are free to use GP tools if you think it's best for
you.  And I will always prefer ID Utils.  My point is that we
shouldn't ignore solutions such as ID Utils, we should include them in
our consideration of what is "fast enough".  We shouldn't consider
only Grep, ripgrep, etc.