bug#65912: 2023-08-31; Fix xref errors when matching in unvisited files

[David Fussner]

Hi.

I've been working on getting the main xref commands working in TeX
files, and ran into a couple of issues.  To reproduce the first one, in
an empty directory put something like the following text into a file,
mwe.ltx (the file extension matters -- please don't try .tex, as that
encounters a separate bug which is better fixed in Emacs itself, I
think):

\documentclass[12pt]{article}
\usepackage{fontspec}
\usepackage{xparse}
\ExplSyntaxOn
\bool_new:N \g__hook_debug_bool
\cs_new_eq:NN \__hook_debug:n \use_none:n
\ExplSyntaxOff
\begin{document}
\thispagestyle{empty}
\end{document}

Then copy mwe.ltx to mwe2.ltx, and visit mwe.ltx without visiting
mwe2.ltx.  Place point on "fontspec" and hit M-?, RTN, then "..." to
choose a search directory, then choose the current one (the default).
Here I see this error:

Font-lock trying to use keywords before setting them up

The end of the backtrace looks like this:

Debugger entered--Lisp error: (error "Font-lock trying to use keywords
before setting th...")
  signal(error ("Font-lock trying to use keywords before setting th..."))
  error("Font-lock trying to use keywords before setting th...")
  
font-lock-compile-keywords((("\\\\\\(?:\\(?:verb\\*?\\)\\)\\(?:\\[[^][]*\\(?:\\[[^][]*\\][^]..."
(1 "\"") (2 ".") (3 "\"")) ("\\(\\\\\\)end
*{\\(?:\\(?:filecontents\\*?\\|verbatim\\*?\\)..." (1 "|" t)) ("^[
\11]*\\\\begin *{\\(?:\\(?:filecontents\\*?\\|verbatim\\..." (1 "|"
t))) t)
  font-lock-fontify-syntactic-keywords-region(1 22)
  #f(compiled-function (start end) #<bytecode 0x835be9f85adc43b>)(1 22)
  syntax-propertize(22)
  xref--collect-matches-1("\\_<fontspec\\_>" "~/src/testing/mwe2.ltx" 2 1 22 0)

The compiled function here is
font-latex--make-syntax-propertize-function, which is a wrapper for
syntax-propertize-via-font-lock, which is a standard value for
syntax-propertize-function, but the in-tree tex-mode.el uses a
different function, and the error doesn't appear there.  Here
syntax-propertize-via-font-lock calls
font-lock-fontify-syntactic-keywords-region, which calls
font-lock-compile-keywords, which errors out because in the temporary
buffer (" *xref-temp*") we haven't yet called font-lock-set-defaults.

The in-tree bibtex.el uses the same syntax-propertize-function but
doesn't error here, because it calls font-lock-set-defaults, with a
comment explaining why, at the end of the definition of bibtex-mode
itself.  If you visit both mwe2.ltx and mwe.ltx, the error won't appear,
because the font-locking mechanism calls font-lock-set-defaults when
setting up the buffer.

I enclose a patch (0001), which puts the required call, with a comment, at
the end of font-latex-set-syntactic-keywords, which seems to work for
all the AUCTeX modes, but there may be a better place for it.

If you apply patch 0001, and make sure font-latex.el is reloaded, you
can then see the next bug: in specific cases, xref-find-references
reports a hit only in visited files, despite there being identical text
in the unvisited one.

To reproduce, move point in mwe.ltx from "fontspec" to
"\__hook_debug:n", then call M-? just as before. You'll only see the hit
in mwe.ltx in the xref buffer, not the one in mwe2.ltx.  The problem is
that the search string called by xref-find-references has the format
"\\_<%s\\_>", so that any search term which doesn't start and end with
chars having either word or symbol syntax won't be returned.  Now,
assuming that parsing is enabled, AUCTeX does the right thing here and
parses this file as containing expl3 constructs, and therefore gives "_"
symbol syntax.  Unfortunately, the style file expl3.el will not be
applied to temporary, non-file-visiting buffers like " *xref-temp*",
because the find-file-hook set at the end of
VirTeX-common-initialization won't be called in such buffers.  The
attached patch (0002) parses and applies styles in the temp buffer, and
only in such non-file-visiting buffers.  It does so both in
LaTeX-common-initialization and in plain-TeX-common-initialization,
which I think covers all the modes.

My profoundest apologies for the long bug report, and thanks,

David.

Emacs  : GNU Emacs 30.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.31, cairo version 1.16.0)
 of 2023-09-11
Package: 2023-08-31

current state:
==============
(setq
 AUCTeX-date "2023-08-31"
 window-system 'x
 LaTeX-version "2e"
 TeX-style-path '("/usr/local/var/auctex" "/home/dfussner/src/auctex/style"
          "/home/dfussner/.emacs.d/auctex/auto"
          "/home/dfussner/.emacs.d/auctex/style" "auto" "style")
 TeX-auto-save t
 TeX-parse-self t
 TeX-master nil
 TeX-command-list '(("TeX"
             "%(PDF)%(tex) %(file-line-error) %`%(extraopts)
%S%(PDFout)%(mode)%' %(output-dir) %t"
             TeX-run-TeX nil
             (plain-tex-mode ams-tex-mode texinfo-mode) :help
             "Run plain TeX")
            ("LaTeX" "%`%l%(mode)%' %T" TeX-run-TeX nil
             (latex-mode doctex-mode) :help "Run LaTeX")
            ("Makeinfo" "makeinfo %(extraopts) %(o-dir) %t"
             TeX-run-compile nil (texinfo-mode) :help
             "Run Makeinfo with Info output")
            ("Makeinfo HTML"
             "makeinfo %(extraopts) %(o-dir) --html %t"
             TeX-run-compile nil (texinfo-mode) :help
             "Run Makeinfo with HTML output")
            ("AmSTeX"
             "amstex %(PDFout) %`%(extraopts) %S%(mode)%' %(output-dir) %t"
             TeX-run-TeX nil (ams-tex-mode) :help "Run AMSTeX")
            ("ConTeXt"
             "%(cntxcom) --once --texutil %(extraopts) %(execopts)%t"
             TeX-run-TeX nil (context-mode) :help "Run ConTeXt once")
            ("ConTeXt Full" "%(cntxcom) %(extraopts) %(execopts)%t"
             TeX-run-TeX nil (context-mode) :help
             "Run ConTeXt until completion")
            ("BibTeX" "bibtex %(O?aux)" TeX-run-BibTeX nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode context-mode)
             :help "Run BibTeX")
            ("Biber" "biber %(output-dir) %s" TeX-run-Biber nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run Biber")
            ("Texindex" "texindex %s.??" TeX-run-command nil
             (texinfo-mode) :help "Run Texindex")
            ("Texi2dvi" "%(PDF)texi2dvi %t" TeX-run-command nil
             (texinfo-mode) :help "Run Texi2dvi or Texi2pdf")
            ("View" "%V" TeX-run-discard-or-function t t :help
             "Run Viewer")
            ("Print" "%p" TeX-run-command t t :help "Print the file")
            ("Queue" "%q" TeX-run-background nil t :help
             "View the printer queue" :visible TeX-queue-command)
            ("File" "%(o?)dvips %d -o %f " TeX-run-dvips t
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Generate PostScript file")
            ("Dvips" "%(o?)dvips %d -o %f " TeX-run-dvips nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert DVI file to PostScript")
            ("Dvipdfmx" "dvipdfmx -o %(O?pdf) %d" TeX-run-dvipdfmx nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert DVI file to PDF with dvipdfmx")
            ("Ps2pdf" "ps2pdf %f %(O?pdf)" TeX-run-ps2pdf nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Convert PostScript file to PDF")
            ("Glossaries" "makeglossaries %(d-dir) %s" TeX-run-command
             nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run makeglossaries to create glossary file")
            ("Index" "makeindex %(O?idx)" TeX-run-index nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run makeindex to create index file")
            ("upMendex" "upmendex %(O?idx)" TeX-run-index t
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run upmendex to create index file")
            ("Xindy" "texindy %s" TeX-run-command nil
             (plain-tex-mode latex-mode doctex-mode ams-tex-mode
              texinfo-mode)
             :help "Run xindy to create index file")
            ("Check" "lacheck %s" TeX-run-compile nil (latex-mode)
             :help "Check LaTeX file for correctness")
            ("ChkTeX" "chktex -v6 %s" TeX-run-compile nil (latex-mode)
             :help "Check LaTeX file for common mistakes")
            ("Spell" "(TeX-ispell-document \"\")" TeX-run-function nil
             t :help "Spell-check the document")
            ("Clean" "TeX-clean" TeX-run-function nil t :help
             "Delete generated intermediate files")
            ("Clean All" "(TeX-clean t)" TeX-run-function nil t :help
             "Delete generated intermediate and output files")
            ("Other" "" TeX-run-command t t :help
             "Run an arbitrary command")
            )
 )

0001-Fix-xref-error-when-matching-in-unvisited-files.patch
Description: Text Data

0002-Fix-xref-error-when-matching-in-unvisited-files.patch
Description: Text Data