Re: Generation of tags for the current project on the fly

[Dmitry Gutov]

On 1/12/18 12:01 PM, Eli Zaretskii wrote:

> Why discard it after the first save?  The tags table is probably still
> very much valid.

Indeed, it's a rough heuristic. I'm aiming for correctness here, not for 
performance.

On the other hand, code navigation and editing are often fairly distinct 
activities, you don't switch between the two too frequently. So waiting 
a second or two when going from the latter to the former shouldn't be 
too terrible.

> I'd not discard it until either of the following
> happens:
>
>    . we fail to find a tag

Not sure about this one. We can make this customizable, of course 
(although the implementation might end up a bit convoluted), but IMO 
it's not good for the default behavior.

Failing to find a tag is a valid result (some identifiers can be absent, 
or defined somewhere else, e.g. in the libraries), and doing a rescan 
each time that happens might be more annoying.

Further, some users will call C-u xref-find-definitions, look for the 
new tag in the completion table, fail to find it there, and simply abort 
without trying the search.

>    . the user visits a tags table explicitly

That's of course, works already.

>    . the user switches to a different project(?)

It's an omission currently, but yes, I fully intend to add this.

> We could offer generating a tags table if we don't find one in the
> tree, instead of generating it automatically.

And then what? Visit it? And make the user to rescan manually every 
time? I'm fine with this as an optional behavior (and it will also be an 
improvement, of course, since generating tags is not exactly trivial for 
new users, and even many not-so-new ones), but I don't want this for the 
default.

> I think this would be a
> better UI and UX, especially given the time it could take to generate
> TAGS (see below).

Sublime Text, Atom and VS Code simply index the project code, AFAIK, 
without extra prompts. I think we should try to show a similar 
experience, even if it's not great for big projects. There are several 
directions we can improve on it, but showing the user that "yes, we can 
find-definition right away" is a good thing.

> > For reference, indexing the Emacs sources takes ~1.1sec here.
>
> Was that with cold cache or warm cache?

Warm, probably. But that's the relevant time, isn't it? We're most 
wondering how long it will take to *reindexing* (because we're 
discussing when to do it). The first indexing will take place anyway.

> "make TAGS" takes about 9 sec here with a warm cache, and this is an
> SSD disk.

'make tags' makes 1 second on my machine, with an NVMe disk.

> On fencepost.gnu.org, a (somewhat slow) GNU/Linux system,
> it took 12 sec with a cold cache and 4 sec with a warm cache.  And
> Emacs is not a large project; I wonder what would happen in larger
> ones, like GCC or glibc.

We can try to somehow detect very large projects, and helpfully offer to 
visit a tags table instead. Anyway, M-x visit-tags-table still works.

> IOW, I don't think this is so fast that we could do that without user
> approval.

The argument here is that if the user called xref-find-definitions, it's 
better to do a (long-ish) scan and show something, instead of failing. 
They always have an option of C-g (we could also catch it and show 
helpful instructions if the process took too long).

> I don't understand why you didn't use the commonly used form:
>
>     find . -name "*.rb" -o -name "*.js" ... | etags -o- -

Because the project API doesn't make this easy. Anyway, generating the 
full list of files is relatively fast in comparison. At most, it took 
like 30% of the whole time (and less in other cases). And we can speed 
it up further independently (e.g. using git ls-files).

> Doing things the way you did raises issues with encoding of file
> names, which could cause subtle problem in rare use cases.

Well, I haven't seen them yet, and don't really understand how they're 
going to happen. But we'll probably fix them, one way or another.

> I think
> using 'find' is also faster.

find is used under the covers. The difference is just that the 
invocations of etags are only happening later.

> More generally, I think doing this that way is not TRT, at least not
> by default.  "make TAGS" in Emacs will produce a much richer tags
> table than your method, because our Makefiles use regexps to augment
> the automatic tagging in etags.  So I think we should first try to
> invoke the TAGS target of a Makefile in the tree, if one exists, and
> only use the naïve command as fallback.

'make tags' is very much specific to Emacs. We can introduce some kind 
of protocol, of course, but my primary goal here is to improve the 
out-of-the-box behavior.

Further, the task will have to write tags to stdout: the current code 
saves the temporary tags file to /tmp, and there are reasons to do that. 
Anyway, that part shouldn't be too hard.

A possible venue for improvement is to somehow derive a multi-TAGS-files 
structure (with their dependencies) from the project information. Still 
thinking about it.