[Orgmode] Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi,

Carsten Dominik wrote:
> On Sep 30, 2010, at 12:20 AM, Sébastien Vauban wrote:
>> Carsten Dominik wrote:
>>> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>>>> Carsten Dominik wrote:
>>>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>>>> Of course, I have many, many files in Org mode. All files I write (or
>>>>>> touch) in fact.
>>>>>>
>>>>>> Of course, I would like to search through my files at some point in
>>>>>> time. The problem is the load-time of my Emacs, now 221 seconds, coming
>>>>>> from 20 seconds before the heavy use of Org...
>>>>>
>>>>> 4 minutes of startup time is entirely unacceptable. And I think you need
>>>>> to identify what is causing this.
>>>>>
>>>>> fontification at display time is standard, I believe.
>>>>
>>>> It seems not, from what I see in the Messages buffer. I really don't have
>>>> the impression of having fiddled with that, really.
>>
>>>>> You must be doing something strange [...] maybe forcing global
>>>>> fontification for each file or so.
>>>>
>>>> To repeat myself, no to that question. But ispell and flyspell are called
>>>> for Org files. Maybe there are interactions?
>>>>
>>>> [Using org-agenda-text-search-extra-files] reduces my load time from 221
>>>> seconds down to 92 seconds. Already a huge diff!
>>>
>>> Still bad though. I am wondering what is causing the fontification
>>> message. I do not get this, so it must be something in your setup. You
>>> should try to find out when this is happening and why.
>>
>> You've seen, from my reply to Matt, why my *Messages* buffer is more
>> verbose than yours. It's on purpose, just to be able to figure out more
>> easily what's going on. Though, I don't have yet...
>>
>>> Also, you might consider to remove (org-agenda-list) from .emacs. I think
>>> it is pretty much always a bad idea to put a command like this into your
>>> startup. Just make it a habit to call it early after starting Emacs.
>>
>> Honestly, that would not change that much. Having to wait 92 seconds at
>> startup, or a long time just a few minutes later has the same impact for
>> me. And: don't try to make me stop using Org ;-))
>
> :-)
>
> The reason why I am suggesting to remove this command is more that it make
> emacs more usable for command line usage etc.

Not convinced yet ;-)

For command-line Emacs, either I use emacsclient or emacs without my config
file, so that's not the problem there.

And the fact is I need my agenda quite early in the process (after firing up
Emacs).


> Furthermore, you are adding quite a bit to font-lock, but I don't see why it
> should be so slow.

Does someone know if the way I use column-marker and the highlighting some
keywords is wrong?


> My Emacs starts up in 4 seconds at most. When I create an agenda, I am
> talking 1 second to maybe 2. I have about 10 files contributing to the
> agenda.

I would love such figures.


>>> It also seems to me you some of the extra packages might be activated
>>> several times in each file. One possible reason could be that you have put
>>> the code to turn them on into several hooks like text-mode-hook,
>>> outline-mode-hook, and org-mode-hook. Turning on org-mode will first run
>>> text-mode-hook, then outline-mode-hook, then org-mode-hook ..... Maybe you
>>> are also calling font-lock-fontify-buffer explicitly in one of your hooks?
>>
>> The only fontification customs that I do are here:
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; special words
>> (setq keywords-critical-pattern
>>      "\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\|
>> [Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff]
>> [Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>> (make-face 'keywords-critical)
>> (GNUEmacs (set-face-attribute 'keywords-critical nil :foreground "red"
>> :background "yellow" :weight 'bold))
>>
>> (setq keywords-org-critical-pattern
>>      "\\(BUGS\\|FIXME\\|XXX\\|[^*] TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm]
>> [Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa]
>> [Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
>>                                  ; smaller subset of keywords for ensuring
>> no conflict with Org mode TODO keywords
>>
>> ;; FIXME Highlighting all special keywords but "TODO" in Org mode is already
>> a
>> ;; good step. Though, a nicer integration would be that "TODO" strings in
>> the
>> ;; headings are not touched by this code, and that only "TODO" strings in
>> the
>> ;; text body would be. Don't know (yet) how to do that...
>> (make-face 'keywords-org-critical)
>> (GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground "red"
>> :background "yellow" :weight 'bold))
>>
>> (setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
>> (make-face 'keywords-normal)
>> (GNUEmacs (set-face-attribute 'keywords-normal nil :foreground "magenta2"
>> :background "yellow"))
>>
>> ;; set up highlighting of special words for proper selected major modes only
>> (dolist (mode '(fundamental-mode
>>                svn-log-view-mode
>>                text-mode))  ; no interference with Org mode (which derives
>> from text-mode)
>>  (font-lock-add-keywords mode
>>                          `((,keywords-critical-pattern 1 'keywords-
>> critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; set up highlighting of special words for Org mode only
>> (dolist (mode '(org-mode))
>>  (font-lock-add-keywords mode
>>                          `((,keywords-org-critical-pattern 1
>> keywords-org-critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; add fontification patterns (even in comments) to a selected major mode
>> ;; *and* all major modes derived from it
>> (defun fontify-keywords ()
>>  (interactive)
>>  (font-lock-add-keywords nil
>>                          `((,keywords-critical-pattern 1 'keywords-
>> critical prepend)
>>                            (,keywords-normal-pattern 1 'keywords-
>> normal prepend))))
>>
>> ;; set up highlighting of special words for selected major modes *and* all
>> ;; major modes derived from them
>> (dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl-
>> mode-hook css-mode-hook
>>                emacs-lisp-mode-hook
>>                html-mode-hook
>>                java-mode-hook
>>                latex-mode-hook lisp-mode-hook
>>                makefile-mode-hook message-mode-hook
>>                php-mode-hook python-mode-hook
>>                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
>>  (add-hook hook 'fontify-keywords))
>> --8<---------------cut here---------------end--------------->8---
>>
>> The goal is to highlight words such as FIXME, WARNING, XXX, TODO in as many
>> files as possible.
>>
>> For Org, a small custom (of the custom) is done in order not to catch all
>> TODO words: the TODO words in headlines (i.e., preceded by `* ') are *not*
>> highlighted.
>>
>> They are well highlighted when found in other portions of (normal) text.
>>
>> Other custom:
>>
>> --8<---------------cut here---------------start------------->8---
>> ;; highlight columns 78 to 80 in some modes
>> (when (try-require 'column-marker)
>>  (dolist (hook '(emacs-lisp-mode-hook
>>                  cperl-mode-hook
>>                  shell-mode-hook
>>                  text-mode-hook
>>                  change-log-mode-hook
>>                  makefile-mode-hook
>>                  message-mode-hook
>>                  texinfo-mode-hook))
>>    (add-hook hook (lambda ()
>>                     (interactive)
>>                     (column-marker-1 78)
>>                     (column-marker-2 79)
>>                     (column-marker-3 80))))
>> --8<---------------cut here---------------end--------------->8---
>>
>> for getting markers in columns 78 to 80, showing me I'm good for a `M-q'...
>>
>> I've done multiple tests, now:
>>
>> - with none of the 2 above sets of customs
>> - with the 1st custom disabled, and the column-marker one enabled
>> - with both enabled
>>
>> Times observed (in seconds):
>>
>> |      | no highlight | column marker only | all highlights |
>> |------+--------------+--------------------+----------------|
>> |      |           63 |                 82 |             91 |
>> |      |           65 |                 84 |             96 |
>> |      |           76 |                100 |            130 |
>> |------+--------------+--------------------+----------------|
>> | mean |           68 |                 89 |            106 |
>>
>> (note that times are really not that stable, while I always launched Emacs
>> in the same conditions, without doing anything else during that time!)
>>
>> So, the above 2 sets of customs have a real impact on the load time.
>>
>> But I'm still unsure how I have to change all of the above snippets of
>> code, in order to keep the same functionality, but without the time delays
>> when Emacs starts up...
>>
>>
>>> Hope some of this helps. If not, let me see you full configuration - maybe
>>> I can spot something else.
>>
>> My .emacs file is there:
>>
>> http://www.mygooglest.com/sva/emacs-init-sva.el
>>
>> Though it's well documented, it's not a tiny one... But very well
>> structured, so quite easy to find stuff related to each other.
>>
>> I'm always interested by any comment...

Best regards,
  Seb

-- 
Sébastien Vauban