Here's the change I was suggesting.
The difference is subtle, but seems like an improvement:
diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index 63e065e696e..581eda0513e 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -1126,7 +1126,9 @@ xref--insert-xrefs
maximize (xref-location-line
(xref-item-location xref)))
for line-format = (and max-line
- (format "%%%dd:" (1+ (floor (log
max-line 10)))))
+ (format
+ #("%%%dd:" 0 4 (face xref-line-number)
5 6 (face shadow))