Re: Asynchronous org-agenda-redo

[Adam Porter]

Ihor Radchenko <address@hidden> writes:

>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>> cooperative, so functions called as threads must yield back to the main
>> thread for Emacs to do anything else before the function returns.
>
> I tried to read the manual, but I clearly misunderstand something.
> The manual says:
>
>>   Currently, thread switching will occur upon explicit request via
>> ‘thread-yield’, when waiting for keyboard input... 
>
> So, except directly calling thread-yield, it should be possible to
> trigger switching the current thread when keyboard input is expected.
> I tried the following demo code:
>
> (defun test ()
>   (let ((a 0))
>     (dotimes (_ 5)
>       (setq a (1+ a))
>       (sleep-for 2)
>       (message "%s" a))))
>
> (progn ;This should return to command loop quickly
>   (make-thread #'test)
>   (message "Executed...")); `eval-last-sexp' here
>
> I can move around the buffer while the progn is running.
> However, it is not the case with `org-agenda-redo' for a reason I do not
> fully understand.

Org Agenda code does not wait for keyboard input; it's busy building the
agenda.  This is the case with most code in Emacs: it's not written to
be asynchronous, and it doesn't return to the main thread until done.
So you can sprinkle yields here and there and maybe be able to move
point around while some code is running, but that will decrease
performance, as well as introducing another level of complexity and
another class of bugs (e.g. what if the user modifies a buffer while the
agenda code is scanning it?).

>> 1.  The process would have to load the same Org buffers, which takes
>>     time, especially in large buffers.  Depending on configuration, it
>>     can take some time, indeed.
>
>> 3.  Ensuring that configuration and state between the main Emacs process
>>     and the separate, agenda-generating process is not necessarily
>>     simple.  Consider as well that if a buffer had unsaved changes,
>>     those would not be readable by the other process, which would lead
>>     to invalid results.  One could force the buffers to be saved first,
>>     but that may not always be desirable, as saving buffers can have
>>     side effects.
>
> Why cannot org-buffer simply be copied into the subordinate process? If
> all be buffer-locals, text properties, and overlays are copied directly
> from the main emacs process, there may be no need to even initialise
> org-mode (the idea is to do something similar to clone-buffer).

AFAIK there exists no way to do such a thing.  Buffers are not designed
to be serialized/deserialized like that.  You could try writing some
Elisp code to do it, but the end result would probably be much slower
than existing agenda code, as well as more difficult to debug.

> The question though is whether buffer-locals + overlays + propertized
> .org files text + org-agenda-buffer copy can be sufficient to make the
> org-agenda-redo run properly. Are there any other buffers, variables,
> or other environment settings used by org-agenda-redo?

As you can see in org-agenda.el, it's complicated.  Remember that an
Emacs process is like a Lisp image, full of state.  The more symbols and
other structures you copy to the async Emacs process (by printing and
reading them as text, remember), the slower it's going to be--and it
will always be slower than not using async.

>> If your agenda buffers are taking too long to refresh, you might
>> consider org-ql's views/saved-searches as an alternative. ...
>
> I know org-ql and I am pretty sure that it will improve performance.
> Actually, if one can make built-in org-agenda asynchronous, org-ql can
> probably use similar approach and become even faster :)

Asynchronous code is not faster; it's generally slower because of
yielding and synchronization.

> I am trying on default org-agenda now mostly because my current config
> is heavily geared towards default agenda and I am not sure if
> refactoring everything to use org-ql will worth it at the end in terms
> of performance. I use too many slow custom skip-functions.

org-ql doesn't use skip functions, just queries.