Re: How to add pseudo vector types

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Thu, 22 Jul 2021 16:11:09 +0200
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
> > That leads to another point. I suspect the memory limit will come
> > before the speed limit, i.e., as the file size increases, the memory
> > consumption will become unacceptable before the speed does. So it is
> > possible that we want to outright disable tree-sitter for larger
> > files, then we don’t need to do much to improve the responsiveness of
> > tree-sitter on large files. And we might want to delete the parse tree
> > if a buffer has been idle for a while. Of course, that’s just my
> > superstition, we’ll see once we can measure the performance.
> 
> Of course those parameters would be configurable on Emacs, but disabling
> TS on a 2MB file because it uses 20MB is way too conservative, IMHO.

Why would we limit ourselves to 20MB?  uint32_t supports upto 4GB.

> Guys, you are speculating too much about minutia and worst-case
> scenarios. (Do we really care about TS not supporting files larger than
> 4GB? I mean, REALLY?)

Yes, we do.  For at least 2 reasons: (a) source code files produced by
programs can be very large; (b) having a feature that fails before you
reach the max size of a buffer Emacs supports is a problem, because it
will cause hard-to-deal-with problems.

Or let me turn the table and ask why we cared to support the largest
possible buffer size when 32-bit systems were the rule?

> I'll rather focus on implementing the thing and optimize later. My bet
> is that a crude implementation would work fine for the 99% of the users
> and be an improvement over what we have now on practically all cases.

This is not a prototype project.  (Or at least I hope it won't end up
being that.)  This is supposed to be the industry-strength code that
core Emacs will use for the years to come to support features which
need language-dependent parsing.  It cannot work correctly only in 99%
of use cases.  So we must assess the limitations seriously and plan
ahead for them.

> BTW, a 10x AST/source-code size ratio is quite reasonable.

It could be, but please don't forget that this is _in_addition_to_ the
"normal" Emacs memory footprint, and that could easily be 1GB and
sometimes several times that.