Re: Is Org really so simple?

[Ihor Radchenko]

> Only philosophy I know is that it is plain text. Is there any official
> philosophy? I have no idea, at least manual does not give me
> references. I cannot find "philosophy", send me references.

You are right. There is no official "philosophy" in org. In my reply I
tried to follow the topic starter's view:

 Texas Cyberthal <texas.cyberthal@gmail.com>:
> By philosophy, I mean the dev consensus on the correct way to do
> things, and coded configuration and usability biases.

According to my experience with org-mode development (I am not talking
about third-party packages here), it is discouraged to change org-mode
towards hiding metadata in org files or store *unique* data (that cannot
be derived from the contents of the org files) related to org-mode
externally (not in org files). It is not official statement, but rather
my impression so far.

> Question is what is meant by database, it can be anything. One can
> save LISP data. Recent files, desktop, eww bookmarks, init.el or
> .emacs file are also all similar databases, there is the underused
> EIEIO with persistent stuff that represent built-in database
> functionality.

Org-mode does not limit user customisations. It does not limit
third-party packages either. Users are free to use any other tools,
store their data anywhere other than org files (why would org force
users to do otherwise?). What I referred to earlier is just the core
org-mode development.

> And people try to do exactly that, people are developing Org in the
> manner to relate objects in Org file to anything else. And they do
> hard work as they do it manually. Relational database speeds up and
> does not tell user to manually hyperlink various relations.

Could you provide some more examples? I do not see how relational
database is different from creating hyperlinks in org. Either way, the
user needs to file an object/headline somewhere into org file, which is
inherently manual.

> I see hard work by many people who try to enhance Org as hierarchical
> knowledge of data because people want to have feature X, but group of
> those enhancements in reality belong to relational databases and not
> to text files.

It is an interesting point. I would be happy if some existing tools
could be reused instead of re-inventing the wheel for org. Do you have
concrete examples where it can be useful? If you have, I encourage you
to bring up a feature request to discuss this possibility with org-mode
devs.

Best,
Ihor

Jean Louis <bugs@gnu.support> writes:

> * Ihor Radchenko <yantar92@gmail.com> [2020-11-23 17:18]:
>   :PROPERTIES:
>   :CREATED:  [2020-11-23 Mon 18:42]
>   :ID:       edebb3e7-e755-4ecc-a5e8-a3353a3f5fd0
>   :END:
>> Dear Jean Louis,
>> 
>> Your description of the database reminds me how org-roam handles the
>> files - it also uses an external database for linking and allows quick
>> incremental search that does not really depend on where the
>> files/headings are stored.
>
> Sounds good, I can see there is graph database used.
>
>> However, what you are talking about is against org-mode philosophy,
>> as I know it.
>
> Only philosophy I know is that it is plain text. Is there any official
> philosophy? I have no idea, at least manual does not give me
> references. I cannot find "philosophy", send me references.
>
> It says "to keep simple things simple". But Org is far far from being
> simple any more. It offers good principles, paradigms and people built
> many enhancements upon those. Speedily it becomes way much more than
> simple.
>
> Headings do not look any more as headings, it looks like pieces of
> code to a person that is new. Properties, tags, clocks, schedule,
> deadline, all that tries to store itself under specific heading. There
> is easily too much of it and things are not simple any more.
>
>> Currently, the dev's stance is that org files must be
>> self-sufficient.
>
> There is no compact principle there practically. Anything is
> possible. That Org files are not practically self-sufficient shows the
> fact that there are 129 Emacs packages in one Org distribution. 
>
>> Org-mode should not depend on external database to manage the org
>> files and operations with them. Everything must be plain text!
>
> Question is what is meant by database, it can be anything. One can
> save LISP data. Recent files, desktop, eww bookmarks, init.el or
> .emacs file are also all similar databases, there is the underused
> EIEIO with persistent stuff that represent built-in database
> functionality.
>
> That Org files are not self-sufficient shows the demand that there is
> almost no Org user who does not have add-ons, packages, modifications,
> configurations.
>
> Would it be really self-sufficient there would be no development going
> on, right?
>
> Babel executions clearly show that Org is not self sufficient and
> depends on number of external software.
>
> I don't mind of philosophy, in fact I would like that philosophy is
> really that what it wanted to be, but that time is over.
>
> I am just pointing out that it is by many means not self sufficient.
>
> Is by default LaTeX export enabled? I think it is. How big is the
> LaTeX package? It is huge, and Org depends on it for export.
>
> Thus Org is far far from being self-sufficient.
>
> Almost every system has GDBM database, if Emacs would have bindings to
> GDBM, there would be so much less of development in general for many
> various packages and Emacs would be expanding faster and easier.
>
> In fact I think that author and initial developers could not predict
> at the time where the Org goes and that speaking of self-sufficient
> and "plain text" only is history.
>
> Before I found out about Org I was using back in time `hnb' in console
> to track various planning tasks. It works nice and simple. That is
> really self sufficient. Org definitely not.
>
> HNB - Hierarchical Notebook
> http://hnb.sourceforge.net/
>
> In the mean time I have created database where I can store TODO,
> Notes, Calls, SMS, People, Invoices, Groups, Mailing Lists and so on,
> and made my own shell and Perl interfaces to it. And I used to manage
> it through: GeDaFe - PostgreSQL Generic Database Interface
> http://gedafe.github.io/doc/gedafe-sql.en.html and this was and is
> hierarchical or better graph knowledge management and relational
> database.
>
> Creating simple table in the database automatically helped me to
> manage that table. It is trivial to create NOTES table with schedule
> date, clock in, TODO or other conditions and tags. The interface is
> just there and automatic to whatever table I create the interface is
> there to add/modify/delete/search/refine records. That is what I would
> say "simple" and keeping things simple and indefinitely extensible
> without modification of software. The fundamental GeDaFe principle I
> would like to try to implement in Emacs. And same database I use for
> web interface I am using also within Emacs.
>
> GeDaFe principle is following:
>
> - define your data (like handling notes, TODO, or executing scripts within 
> notes)
>
> - work and forget about any underlying functions,
>   add/create/delete/modify/index or search, make reports with simple exports
>
> - expand with more definitions of data when necessary (like add
>   various properties, or other data tables, like contacts, invoices,
>   etc.) and repeat the process.
>
> Org also shows that it does not hold "Notes" only, it holds more than
> that, I have written average book size technical documents with
> it. Only just one part of the document is printed from one single node
> that belongs to single project among many. People use such documents
> on the ground. My use case is not for simple notes.
>
> A node in a tree can be anything, and Org enhancements develop by that
> principle. For example there is org-contacts where nodes are meant to
> be "People". Such development is so much hard coded.
>
> Simpler would be to define the type of nodes and work by their
> types. One could need just one type table and one node table and that
> is about all.
>
> The type table could say:
>
> - this node is heading
> - or, this node is text under heading
> - or, this node is paragraph among many others
> - or, this node is hyperlink to WWW URI
> - or, this node is hyperlink to file
> - this node is movie to play
> - this type is PDF to be opened on page 3
> - this one is really note
> - this one is note but with action assigned like TODO, etc.
>
> Nodes could have its properties defined like for anything. Nodes can
> reference its parent nodes. Node can be parent to any heading.
>
> Once such 2 tables are defined magic starts to take place, it becomes
> its subtree with all kinds of node types including Babel execution and
> similar. Meta data is meta, it can be updated but need not be
> visible. Computer is handling meta data.
>
> Node can be a page in a subtree that becomes a website.
>
> Node can be object for person or company, or just anything.
>
> I am currently using my nodes to quickly research various subjects by
> using that type of dynamic knowledge repository.
>
> Org file is dynamic knowledge repository.
>
> About Dynamic Knowledge Repositories (DKR)
> https://www.dougengelbart.org/content/view/190/163/
>
> Then around 2015 I have discovered Cherrytree and have made bunch of
> notes with it, and that is self-sufficient one program that keeps
> simple things simple as it is much more simpler than Org. It offers a
> visual interface to all features related to the knowledge management
>
> Cherrytree - hierarchical note taking application with rich text and syntax 
> highlighting
> https://www.giuspen.com/cherrytree/
>
> TAGS in Cherrytree are automatic if I remember well, TAG is simply
> name of the node. If I call node TODO, then nodes under are simply
> TODO items. 
>
> Later I discovered there is something similar in Emacs so I started
> with Org and use it mostly for instruction writing and project
> management. And I find many options over kill for me. On the other
> hand my usage of Org would be overkill for somebody, so it depends of
> viewpoints.
>
> In general I was always using headings and subheadings, trees, lists, notes 
> by using text.
>
> Somewhere 2004 I started using Markdown one among first as I found it
> simpler than ASCIIDOC and M4, but not as perfect.
>
> 2020 and 2021 I like to raise the level of dynamic knowledge
> journaling to the meta level where I think less about underlying
> functions in software.
>
> That experience also tells that Org is definitely not simple.
>
> Among hnb, GeDaFe database approach, Cherrytree and Org mode, for
> "keeping things simple" like note taking and TODO items, project
> management, Cherrytree was the best for me.
>
> Among all those for keeping complex things simple the database
> approach is the best. Of course that I use database within Emacs and
> it is not visible to user what it is. Database allows simultaneous
> operation by several people on distance and that is the groupware
> feature as envisioned by Doug Engelbart.
>
> For document writing and nice formatting with LaTeX export, Org mode
> is the best personally.
>
> For notes, database based notes are best as only so I have relations
> between the note and other objects. With 200,000 contacts in the
> database which I can quickly access and assign notes to them, how
> would that work with Org? Hardly. Notes are related to people, to
> projects, plans, opportunities, research subjects, companies and so
> on. There is no simple way in Org mode to relate one note to multiple
> other related subjects.
>
> And people try to do exactly that, people are developing Org in the
> manner to relate objects in Org file to anything else. And they do
> hard work as they do it manually. Relational database speeds up and
> does not tell user to manually hyperlink various relations.
>
> I have sent thousands of tasks to people by using this function
> below. And I had to define for each Org file who are "assignees" for
> that Org file. And I have like 50 assignees, so I need to copy and
> paste their nick names or identifiers into the Org file. There it
> comes the attribute of being "self-sufficient", it becomes
> self-sufficient because I had to define all assignees into that
> specific file, but I find it tedious and not useful.
>
> (defun rcd/org-send-assigned-task ()
>   "Sends assigned task to designated individual as Org"
>   (interactive)
>   (let* ((member-data (rcd-org-extract-assigned-member-email-data))
>        (id (if member-data (first member-data) nil))
>        (signature (if (equal (type-of (symbol-value (fifth member-data))) 
> 'cons)
>                       (third (symbol-value (fifth member-data))) ""))
>        (file (rcd-org-subtree-to-file signature))
>        (subject (rcd/org-find-headline))
>        (esubject (escape-% subject))
>        (ask (unless id (y-or-n-p "No assignment found. Do you want to send it 
> by email?")))
>        (name (if id (third member-data)))
>        ;; (name (if ask (read-from-minibuffer "Name:") name))
>        (voice (format "The task '%s' is being sent to '%s'" subject name))
>        (email (if id (if (equal (type-of (fourth member-data)) 'cons)
>                          (car (fourth member-data))
>                        (fourth member-data))))
>        (email (if ask (cf-search-email (read-from-minibuffer "Search for 
> email: ")) email))
>        (really (y-or-n-p (format "Do you really want to send it to: %s?" (if 
> ask email name)))))
>     (if (and really (or id ask))
>       (if (string-match "@" email)
>       (progn
>         ;; (message (escape-% subject))
>         (speak voice)
>         (mutt-send-file name email esubject file))
>       (message "No email specified"))
>       (message "Aborted sending."))))
>
>> Moreover, some devs are even reluctant to hide metadata (like unique
>> ID implemented in org-id module) from users (which is possible and
>> also partially implemented).
>
> I hope that I have demonstrated that one who develops could review
> several various paradigms and learn more. I am fine with any decision
> of design for Org mode as I use it as it is and I have for me other
> ways of managing data. I am not sure how much those features have been
> discussed to say that hiding meta data is good or not good. It is
> better to discuss and find what is more useful.
>
> What I can see is that people complain for speed and they build
> extensions that help with it. Extensions are external while built-in
> Org does not keep up with the dynamic how people expect it to be.
>
> For example I would expect Org to be very simple, very very simple, I
> would rewind it back to its roots. Somebody else would like
> complexities. Currently I can see that Org is not made for
> complexities. It is better to unwind the development and make Org in
> core very basic and speedy and let people enhance it with external
> packages. In general it is better to remain simple. Even that may not
> be possible any more.
>
> I see hard work by many people who try to enhance Org as hierarchical
> knowledge of data because people want to have feature X, but group of
> those enhancements in reality belong to relational databases and not
> to text files. Developers wish to accommodate various people and yet
> by doing so they develop it into complex software. (129 .el packages
> for one Emacs mode!?)
>
> Among those one shall read the Doug Engelbart's paradigms, then
> especially if one is developer of system of data management that many
> people use, one shall explore other paradigms, including various
> approaches to data handling. And overall one shall keep it simple as
> it is main fundament of Org to be simple, while practical fact remains
> that it is not anymore simple, not at all. 
>
> I remember back in time with BASIC programming language that it had
> also something like a built-in database that one could put on the
> bottom of the file. Then there is also Perl's __DATA__ that is placed
> straight into the file and one could also place images and other
> stuff. Maybe the meta data could be kept this way in just one heading
> named Meta data, and this heading would not be exported, it could be
> hidden from the user and it could contain the database necessary for
> single Org file. By pressing key to show properties one could see
> properties or simply make them hidden. By making a copy of subtree the
> metadata would also copy as usual. By exporting, one could make Org
> without meta data, and I like using this information as I like sending
> Org headings without personal meta data to third parties.