Re: [O] How to make agenda generation faster

[Ihor Radchenko]

>   I've thought about this for a while.  It seems to me that the issue is
>   that Org buffers are, of course, plain-text buffers.  There is no
>   persistent, in-memory representation other than the buffer, so whenever
>   Org needs structured/semantic data, it must parse it out of the buffer,
>   which is necessarily rather slow.  If there were a way to keep an
>   outline tree in memory, parallel to the buffer itself, that would allow
>   operations like search, agenda, etc. to be greatly sped up.

FYI
A while ago I saw some cache implementation in org-element.el.
Take a look at org-element--cache variable definition and the code
below.

````
(defvar org-element--cache nil
  "AVL tree used to cache elements.
Each node of the tree contains an element.  Comparison is done
with `org-element--cache-compare'.  This cache is used in
`org-element-at-point'.")
````

Best,
Ihor


Adam Porter <address@hidden> writes:

> Nicolas Goaziou <address@hidden> writes:
>
>>> my understanding is that code that runs with lexical-binding enabled
>>> is generally faster.
>>
>> Not really. But it's certainly easier to understand since it removes one
>> class of problems.
>
> From what I've read, the byte-compiler can optimize better when
> lexical-binding is used.
>
>> Instead of re-inventing the wheel, or putting efforts into a
>> wheel-like invention, wouldn't it make sense to actually work on Org
>> Agenda itself?
>>
>> So again, wouldn't it be nice to think about Org Agenda-ng?
>
> As a matter of fact, what's now called org-ql-agenda was originally
> called org-agenda-ng.  I factored org-ql out of it and realized that it
> should probably be its own, standalone package.  Then I renamed
> org-agenda-ng to org-ql-agenda, so I could reasonably keep them in the
> same repo, and because I don't know if I will ever develop it far enough
> to be worthy of the name org-agenda-ng.  It started as an experiment to
> build a foundation for a new, modular agenda implementation, and maybe
> it could be.
>
>> I didn't look closely at org-ql, but I had the idea of splitting the
>> Agenda in two distinct parts. One would be responsible for collecting,
>> possibly asynchronously, and caching data from Org documents. The other
>> one would provide a DSL to query and display the results extracted from
>> the output of the first part. The second part could even be made generic
>> enough to be extracted from Org and become some part of Emacs.
>> Displaying filtered data, maybe in a timeline, could be useful for other
>> packages. Unfortunately, I don't have time to work on this. Ah well.
>
> I've thought about this for a while.  It seems to me that the issue is
> that Org buffers are, of course, plain-text buffers.  There is no
> persistent, in-memory representation other than the buffer, so whenever
> Org needs structured/semantic data, it must parse it out of the buffer,
> which is necessarily rather slow.  If there were a way to keep an
> outline tree in memory, parallel to the buffer itself, that would allow
> operations like search, agenda, etc. to be greatly sped up.
>
> But how would that work in Emacs?  Theoretically, we could write some
> code, applied on self-insert-command, to update the "parallel tree
> structure" as the user manipulates the plain-text in the buffer
> (e.g. add a new node when the user types a "*" to create a new heading),
> and also apply it to functions that manipulate the outline structurally
> in the buffer.  But, of course, that sounds very complicated.  I would
> not relish the idea of debugging code to keep a cached tree in sync with
> a plain-text buffer outline.  :)
>
> Besides that, AFAIK there would be no way to do it asynchronously other
> than calling out to a child Emacs process (because elisp is still
> single-threaded), printing and reading the data back and forth (which
> would tie up the parent process when reading).  Maybe in the future
> elisp will be multithreaded...
>
> Anyway, org-ql tries to do some of what you mentioned.  It does
> rudimentary, per-buffer, per-query caching (as long as the buffer is not
> modified, the cache remains valid), which helps when there are several
> Org files open that are referred to often but not as often modified.
> And the query and presentation code are separated (org-ql and
> org-ql-agenda).
>
> I don't know how widely it's used, but the repo is getting some regular
> traffic, and I'm using it as the backend for my org-sidebar package.
> I'd be happy if it could be made more generally useful, or if it could
> be helpful to Org itself in some way.  Contributions are welcome.
>
>

signature.asc
Description: PGP signature