bug#66759: 30.0.50; Flymake (with Eglot) error cleaning up old overlay

[João Távora]

Richard Copley <rcopley@gmail.com> writes:

> I'm afraid I'm unable to consistently reproduce this error. I hope you
> can see the issue and devise a testcase from the following
> description.

Thanks very much for this report.  This problem could be the same as
https://github.com/joaotavora/eglot/discussions/1311, at least it its
most recent iteration.

Anyway, I think your analysis of the code is excellent and your
conjecture (for at least one possible cause of this problem) is very
promising.  That "don't bother with invalid bounds" was introduced
recently:

   commit 8b1947ffdd9d9eae26a308f0abaac45e06baac22
   Author: João Távora <joaotavora@gmail.com>
   Date:   Thu Sep 21 00:03:32 2023 +0100
    
       Flymake: more fixes to flymake--highlight-line
       
       Make it robust to diagonstics with invalid bounds.
       
       * lisp/progmodes/flymake.el (flymake--highlight-line): Robustify.
    
   diff --git a/lisp/progmodes/flymake.el b/lisp/progmodes/flymake.el
   --- a/lisp/progmodes/flymake.el
   +++ b/lisp/progmodes/flymake.el
   @@ -781,1 +782,5 @@
   -    (setq ov (make-overlay end beg))
   +    (when (= (overlay-start ov) (overlay-end ov))
   +      ;; Some backends report diagnostics with invalid bounds.  Don't
   +      ;; bother.
   +      (delete-overlay ov)
   +      (cl-return-from flymake--highlight-line nil)):


And indeed the flymake--diag-overlay slot is not filled in when we get
this early return.  And indeed the overlays considered for deletion are
the ones stored in the "state" map, meaning everything the backend
reported.

So maybe this patch is all that's needed:

diff --git a/lisp/progmodes/flymake.el b/lisp/progmodes/flymake.el
index b27e6527f81..9be40499d37 100644
--- a/lisp/progmodes/flymake.el
+++ b/lisp/progmodes/flymake.el
@@ -809,6 +809,7 @@ flymake--highlight-line
                         (flymake--diag-orig-end e))
                   (flymake--delete-overlay eov)))
     (setq ov (make-overlay beg end))
+    (setf (flymake--diag-overlay diagnostic) ov)
     (when (= (overlay-start ov) (overlay-end ov))
       ;; Some backends report diagnostics with invalid bounds.  Don't
       ;; bother.
@@ -863,7 +864,6 @@ flymake--highlight-line
     (overlay-put ov 'evaporate t)
     (overlay-put ov 'flymake-overlay t)
     (overlay-put ov 'flymake-diagnostic diagnostic)
-    (setf (flymake--diag-overlay diagnostic) ov)
     ;; Handle `flymake-show-diagnostics-at-end-of-line'
     ;;
     (when flymake-show-diagnostics-at-end-of-line


There's a fair chance this fixes the bug effectively, but even if it
doesn't, it is nevertheless a solid change, so I've pushed it and bumped
the Flymake ELPA package version.

Please keep an eye out of this bug.

What language server are you using with Eglot btw?

> A possible fix is to check if `flymake--highlight-line' created an
> overlay before inserting a diagnostic into `flymake--state-diags',
> in phase 2.

This could also work, but is slightly more complex.  And it would
destroy the invariant that that list contains every "domestic"
diagnostic reported by the backend (even invalid ones).

João