Perhaps Yuan has some further ideas. There are some strong oddities here:
- Some time into debugging and repeating the benchmark again and again,
I get the "Pure Lisp storage overflowed" message. Just once per Emacs
session. It doesn't seem to change much, so it might be unimportant.
- The profiler output looks like this:
18050 75% - font-lock-fontify-syntactically-region
15686 65% - treesit-font-lock-fontify-region
3738 15% treesit--children-covering-range-recurse
188 0% treesit-fontify-with-override
- When running the benchmark for the first time in a buffer (such as
ruby.rb), the variable treesit--font-lock-fast-mode is usually changed
to t. In one Emacs session, after I changed it to nil and re-ran the
benchmark, the variable stayed nil, and the benchmark ran much faster
(like 10s vs 36s).
In the next session, after I restarted Emacs, that didn't happen: it
always stayed at t, even if I reset it to nil between runs. But if I
comment out the block in treesit-font-lock-fontify-region that uses it
;; (when treesit--font-lock-fast-mode
;; (setq nodes (treesit--children-covering-range-recurse
;; (car nodes) start end (* 4 jit-lock-chunk-size))))
and evaluate the defun, the benchmark runs much faster again: 11s.
(But then I brought it all back, and re-ran the tests, and the variable
stayed nil that time around; to sum up: the way it's turned on is
unstable.)
Should treesit--font-lock-fast-mode be locally bound inside that
function, so that it's reset between chunks? Or maybe the condition for
its enabling should be tweaked? E.g. I don't think there are any
particularly large or deep nodes in ruby.rb's parse tree. It's a very
shallow file.