Re: [emacs-humanities] Paper Zettelkasten safety [was: Why Emacs-humanities?]

[Jean Louis]

* Ihor Radchenko <yantar92@gmail.com> [2020-12-24 01:10]:
> Göktuğ Kayaalp <self@gkayaalp.com> writes:
> 
> > Tangentially, I’m curious about this Zettelkasten thing.  I’ve looked
> > into it, quite a bit, but I don’t get the hype.  Isn’t it pretty much
> > hypertext with some organisation and automated backlinks?
> > Does it really achieve e.g. anything plain Org + Deft/occur/isearch
> > can’t?
> 
> Yes, it is - when you just look at the notes/cards produced by someone
> else. And Zettelkasten can be implemented in plain org (that's what I
> do).
> 
> However, a big part of Zettelkasten is not about implementation. If you
> read the book [1] describing Zettelkasten method, you will see that big
> part of the method is specific workflow. Do you regularly dedicate some
> time to interlink every headline you have to other headlines?

Good point. Assigning relations is most important part of it. What I
do is usually to add a new data object which I call rather "object"
because it is not necessarily an object in the sense of a note, though
all of it could be said to be some kind of notes. After adding the
object I may decide about its type or use. Maybe it is of type
Asciidoctor to be processed with the same named program, maybe it is
for publishing, maybe it is private, or maybe it is of type File, or
image that would be opened with special image viewer, or Note, Task,
maybe Message-ID or similar. After assigning the type comes the
relation to people, organizations, groups, and assignments to people,
organizations, groups. An assignment defines who is responsible to
handle that object, especially when it is a task. Quick and easy
reminding by email, SMS, automated phone call is a must. Then comes
any timestamp, like SCHEDULED, MEANWHILE, etc. that may also have its
description, those are like sub-notes about the note. 

Thining about unique ID is not there. There is unique ID automatically
assigned by the database, there is automatically assigned "date
created", at any modification there is "date modified", at any
creation or modification there is "user that created it" and "user
that modified it" as in a collaborative system multiple users can work
on the same object.

By all means I am in agreement with the Zettelkasten method by
principle to be one of most useful. 

One has to think that nice index always brings order. And in the
computer any attributes becomes even easier findable.

Both ways, the paper should be easily converted to computer with some
reasonable effort like in case of few thousands cards it should be
easily digitalized within few days of work, and computer notes should
be easily converted into a paper report so that they could eventually
form a Zettelkasten method with references if such would be
printed. Printed documents may serve as much more redundant backup of
the true information than a hard disk.

> Deliberately keeping notes unorganised (at the beginning) encourages
> people to do the linking because there is no internal feeling that
> the note can be easily found later.

I understand the idea and by sharing my personal experience, the
unlinked notes do not encourage me to link them just because they are
unlinked. If note is not related to anything it will remain
unrelated. If note is related to a person, group, organization,
business, opportunity, other objects, other people, etc. then it will
be related, it is kind of crying to get related to it as only so it
builds full profile of those related objects as well. A missing
relation in a note is automatically a missing relation in other
referenced objects.
 
But I don't think that deliberately keeping notes unorganized is good
thing, quite contrary, there shall be categories, parent nodes or
similar where new nodes should be placed. For WWW bookmarks I have one
category where everything goes, but by using filter I can choose or
move bulk of by subject related bookmarks to specific WWW bookmark
sub-category.

If I am research a subject X, I can quickly switch to different
category where WWW bookmarks will be sorted by a single click. That
way I need not deliberately keep it unorganized, rather deliberately
keep it organized.

Though I do understand fully your idea as it works for me the same, I
can actually put anyting into the Dynamic Knowledge Repository without
problem just becauset things get well related to each other. Relations
are there and jumping from one to other can be blazing fast, similar
to xref in Emacs.

> The same can be achieved without using cards or small files, but the
> idea to have the habit of thinking about relations between notes and
> summarising the current knowledge on specific topics is probably the
> most important takeaway of Zettelkasten method. I recommend reading
> the book I referenced and try to find the useful ideas. Specific
> software implementations are probably much more a question of
> preference.

I can fully understand the idea and support it. 

> > In general it feels like a somewhat rigid system with some
> > serious technical debt. Lots of little files, a server, non-Org tags,
> > etc. 

I find rigid systems better, that is systems with constrains, where
attributes, tags, categories have all their places where they belong
without any kind of ambiguity. Is either correct or correctly
placed. There cannot be any incorrect piece of information or
mistakenly written or vague.

> What exactly makes it superior to just putting notes in a couple >
> files, and using search and manually linking to stuff?

Though not related to original discussion, what makes the system
superior is rigidness, structure, constraints, where things belong
where they should, and that is the SQL database or other kind of
collaborative query database.

Statistics form my system may show number of notes, majority is PDF by
page number, as I started making it for research of specific
subjects. It is pretty easy to index PDF files in the database or any
other type of a note, some are not indexable. 

            Object            | Count 
------------------------------+-------
 PDF by Page Nr.              | 14439
 WWW                          |  4463
 Set ➾                        |  1305
 PDF                          |   730
 Note                         |   380
 YouTube Video                |   158
 Task                         |   142
 Emacs Lisp Hyperlink         |    70
 YouTube Video at exact time  |    63
 File                         |    54
 FOLLOW-UP                    |    54
 Video                        |    52
 Asciidoctor                  |    43
 Markdown                     |    38
 Local File                   |    28
 MPV play video at exact time |    23
 Org                          |    21
 Directory                    |    20
 EPUB                         |    18
 URL for Image                |    17
 HTML Hyperdocument           |    14
 Image                        |    13
 DJVU                         |    13
 Media                        |     9
 Message-ID                   |     8
 Launch Program               |     6
 Quote                        |     4
 Enriched                     |     4
 HyperScope ID                |     3
 Shell Command                |     3
 Self                         |     3
 Emacs Lisp                   |     2
 Org Heading                  |     2
 Xournalpp                    |     1
 txt2tags                     |     1
 Maff                         |     1
 DISEASE                      |     1
 PostgreSQL                   |     1
 ODT                          |     1
 Kotl (Hyperbole Outline)     |     1
 SUCCESS                      |     1
 Info Node                    |     1
 Meeting                      |     1
(43 rows)

Search by using the PostgreSQL full text search function is blazing
fast. I think that database fits into the memory, so maybe it is all
cached and that is why searching is fast.

Hyperscope full text search with PostgreSQL
https://hyperscope.link/3/6/7/6/8/Hyperscope-full-text-search-with-PostgreSQL-36768.html

Mastering PostgreSQL Tools: Full-Text Search and Phrase Search - Compose 
Articles
https://compose.com/articles/mastering-postgresql-tools-full-text-search-and-phrase-search/

The indexing function is here, it may look complicated at first sight,
but for any kind of tables in the database it can be easily
constructed within 10 minutes. In the long run it offers full text
search without delays.

(defun hyperscope-update-tokens (&optional id)
  "Update tokens for PostgreSQL database full text search.

Optional ID will update single hyperdocument.

Interactively called, a prefix arg 0 will mean to update the
whole database.

Any other prefix arg means to update only hyperdocuments without
tokens."
  (interactive)
  (when-tabulated-id "hyobjects"
    (let* ((where (cond ((and current-prefix-arg 
                              (not (listp current-prefix-arg))
                              (= current-prefix-arg 0))
                         "")
                        ((bound-and-true-p current-prefix-arg) " WHERE 
hyobjects_tokens IS NULL ")
                        (t (format " WHERE hyobjects_id = %s" id))))
           (_ (message "Updating not updated tokens"))
           (sql (format "UPDATE hyobjects SET hyobjects_tokens =
           to_tsvector(concat_ws(' ', (SELECT hyobjectypes_name
           FROM hyobjectypes WHERE hyobjectypes_id = hyobjects_id),
           hyobjects_name, coalesce(hyobjects_link,''), 
coalesce(hyobjects_description,''),
           coalesce(hyobjects_text,''), coalesce(hyobjects_internal,''),
           coalesce((SELECT string_agg(tags_name,' ') FROM tags, hytags 
           WHERE hytags_tags = tags_id AND hytags_hyobjects = 
hyobjects_id),''), 
           (SELECT hyobjects_name FROM hyobjects WHERE
           hyobjects_id = hyobjects_parent),
           coalesce(get_accounts_name(hyobjects_accounts),''),
           coalesce(get_full_contacts_name(hyobjects_people),''),
           coalesce(hyobjects_authorname,''),
           coalesce(get_full_contacts_name(hyobjects_author),''),
           coalesce(get_full_contacts_name(hyobjects_assignedperson),''), 
coalesce((SELECT
           actionstatuses_name FROM actionstatuses WHERE
           actionstatuses_id = hyobjects_actionstatuses),''), coalesce((SELECT
           areas_name FROM areas WHERE areas_id = hyobjects_areas),''),
           coalesce(hyobjects_slug,''), coalesce(hyobjects_report,''),
           coalesce(hyobjects_tempdoc,''))) %s" where)))
      (message "Updating PostgreSQL tokens for table `hyobjects'%s" where)
      (rcd-sql sql cf-db))))



The search function is this one:

(defun hyperscope-search-ts-query (&optional query exclude-set)
  (interactive)
  (let* ((query (or query (read-from-minibuffer
                           "Text search query: " nil nil nil 
'hyperscope-search-history)))
         (query (split-string query))
         (query (string-join query " & "))
         (where (if exclude-set (format "AND hyobjects_parent != %s" 
exclude-set) ""))
         (sql (format "SELECT hyobjects_id, hyobjects_name, 
ts_rank(hyobjects_tokens,'%s') AS rank FROM hyobjects WHERE hyobjects_tokens @@ 
to_tsquery('%s') %s ORDER BY rank DESC" query query where))
         (id-list (rcd-sql-list sql hs-db))
         (id-list (mapcar (lambda (item) (car item)) id-list)))
    (hyperscope nil nil nil nil nil id-list)))

That is about all. New objects added are indexed, they can be
reindexed upon modification, search is quick accurate, so far I never
missed it. And I cannot know if I did miss some query. Better said, so
far I always find what I need. There is no waiting time, it appears
instant. 


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/