Re: Asynchronous org-agenda-redo

[Ihor Radchenko]

>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.

> Anyway, I will try to throw yields into agenda code just to check how
> bad the performance can degrade.

With the following code, org-agenda-redo runs for 21 second on my
system, while without threads it is 16 seconds. However, emacs remains
responsive during rebuilding agenda!

(define-advice org-agenda-redo (:around (oldfun &optional all) make-async)
  (make-thread (lambda () (funcall oldfun all)) "org-agenda-redo"))
(define-advice org-agenda-skip-eval (:around (oldfun form) make-async)
  (thread-join (make-thread (lambda () (funcall oldfun form)) 
"org-agenda-skip-eval")))

The problem, of course, is that touching agenda buffer and org buffers
may be risky while org-agenda-redo is running.
Wondering if it is possible to block user commands during that time. 

Best,
Ihor


Ihor Radchenko <address@hidden> writes:

>> Org Agenda code does not wait for keyboard input; it's busy building the
>> agenda.  This is the case with most code in Emacs: it's not written to
>> be asynchronous, and it doesn't return to the main thread until done.
>> So you can sprinkle yields here and there and maybe be able to move
>> point around while some code is running, but that will decrease
>> performance, as well as introducing another level of complexity and
>> another class of bugs (e.g. what if the user modifies a buffer while the
>> agenda code is scanning it?).
>
> Thanks for the explanation.
>
>> AFAIK there exists no way to do such a thing.  Buffers are not designed
>> to be serialized/deserialized like that.  You could try writing some
>> Elisp code to do it, but the end result would probably be much slower
>> than existing agenda code, as well as more difficult to debug.
>
> Yeah. Even re-initialisation of, for example, overlays in org buffer is 
> likely to take too much time.
>
>> As you can see in org-agenda.el, it's complicated.  Remember that an
>> Emacs process is like a Lisp image, full of state.  The more symbols and
>> other structures you copy to the async Emacs process (by printing and
>> reading them as text, remember), the slower it's going to be--and it
>> will always be slower than not using async.
>
>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.
>
> I see now that generating agenda in separate process will cause too much
> overheads.
> Anyway, I will try to throw yields into agenda code just to check how
> bad the performance can degrade.
>
>> org-ql doesn't use skip functions, just queries.
>
> Skip functions are essentially used-defined queries as soon as the
> queries are tested against every headline.
> I can rewrite my skip functions into queries, but I don't expect much
> improvement since org-ql seems to use org-entry-get, which is the main
> performance bottleneck for my agenda generation.
>
> Best,
> Ihor
>
> adam Porter <address@hidden> writes:
>
>> Ihor Radchenko <address@hidden> writes:
>>
>>>> Be sure to read the Emacs Lisp manual regarding threads.  They are
>>>> cooperative, so functions called as threads must yield back to the main
>>>> thread for Emacs to do anything else before the function returns.
>>>
>>> I tried to read the manual, but I clearly misunderstand something.
>>> The manual says:
>>>
>>>>   Currently, thread switching will occur upon explicit request via
>>>> ‘thread-yield’, when waiting for keyboard input... 
>>>
>>> So, except directly calling thread-yield, it should be possible to
>>> trigger switching the current thread when keyboard input is expected.
>>> I tried the following demo code:
>>>
>>> (defun test ()
>>>   (let ((a 0))
>>>     (dotimes (_ 5)
>>>       (setq a (1+ a))
>>>       (sleep-for 2)
>>>       (message "%s" a))))
>>>
>>> (progn ;This should return to command loop quickly
>>>   (make-thread #'test)
>>>   (message "Executed...")); `eval-last-sexp' here
>>>
>>> I can move around the buffer while the progn is running.
>>> However, it is not the case with `org-agenda-redo' for a reason I do not
>>> fully understand.
>>
>> Org Agenda code does not wait for keyboard input; it's busy building the
>> agenda.  This is the case with most code in Emacs: it's not written to
>> be asynchronous, and it doesn't return to the main thread until done.
>> So you can sprinkle yields here and there and maybe be able to move
>> point around while some code is running, but that will decrease
>> performance, as well as introducing another level of complexity and
>> another class of bugs (e.g. what if the user modifies a buffer while the
>> agenda code is scanning it?).
>>
>>>> 1.  The process would have to load the same Org buffers, which takes
>>>>     time, especially in large buffers.  Depending on configuration, it
>>>>     can take some time, indeed.
>>>
>>>> 3.  Ensuring that configuration and state between the main Emacs process
>>>>     and the separate, agenda-generating process is not necessarily
>>>>     simple.  Consider as well that if a buffer had unsaved changes,
>>>>     those would not be readable by the other process, which would lead
>>>>     to invalid results.  One could force the buffers to be saved first,
>>>>     but that may not always be desirable, as saving buffers can have
>>>>     side effects.
>>>
>>> Why cannot org-buffer simply be copied into the subordinate process? If
>>> all be buffer-locals, text properties, and overlays are copied directly
>>> from the main emacs process, there may be no need to even initialise
>>> org-mode (the idea is to do something similar to clone-buffer).
>>
>> AFAIK there exists no way to do such a thing.  Buffers are not designed
>> to be serialized/deserialized like that.  You could try writing some
>> Elisp code to do it, but the end result would probably be much slower
>> than existing agenda code, as well as more difficult to debug.
>>
>>> The question though is whether buffer-locals + overlays + propertized
>>> .org files text + org-agenda-buffer copy can be sufficient to make the
>>> org-agenda-redo run properly. Are there any other buffers, variables,
>>> or other environment settings used by org-agenda-redo?
>>
>> As you can see in org-agenda.el, it's complicated.  Remember that an
>> Emacs process is like a Lisp image, full of state.  The more symbols and
>> other structures you copy to the async Emacs process (by printing and
>> reading them as text, remember), the slower it's going to be--and it
>> will always be slower than not using async.
>>
>>>> If your agenda buffers are taking too long to refresh, you might
>>>> consider org-ql's views/saved-searches as an alternative. ...
>>>
>>> I know org-ql and I am pretty sure that it will improve performance.
>>> Actually, if one can make built-in org-agenda asynchronous, org-ql can
>>> probably use similar approach and become even faster :)
>>
>> Asynchronous code is not faster; it's generally slower because of
>> yielding and synchronization.
>>
>>> I am trying on default org-agenda now mostly because my current config
>>> is heavily geared towards default agenda and I am not sure if
>>> refactoring everything to use org-ql will worth it at the end in terms
>>> of performance. I use too many slow custom skip-functions.
>>
>> org-ql doesn't use skip functions, just queries.
>>
>>
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong 
University, Xi'an, China
Email: address@hidden, address@hidden