Re: using finalizers

[Tomas Hlavaty]

On Sun 02 Jan 2022 at 10:33, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Tomas Hlavaty <tom@logand.com>
>> Cc: rudi@constantly.at, emacs-devel@gnu.org
>> Date: Sun, 02 Jan 2022 08:53:31 +0100
>> 
>> On Sun 02 Jan 2022 at 08:54, Eli Zaretskii <eliz@gnu.org> wrote:
>> > One common paradigm for such use cases is to use a callback: the
>> > function that traverses some collection of objects calls that callback
>> > for every object it finds.
>> 
>> There is no such thing in directory-files.
>
> It should be easy to add, if we wanted to.  Or we could implement a
> separate set of primitives using that, if it were deemed useful.
>
> I fail to see how what's in Emacs now is of any essence for the
> purposes of this discussion.  What exactly are you getting at, and how
> is what is in Emacs today relevant?  Are you saying that Emacs
> couldn't possibly provide such primitives due to some inherent
> limitation?  If so, please explain why you think so, because I don't
> think I agree.

I was replying to Rudolf about the use-cases where stack discipline can
have visible negative consequences and demonstrating it with a few
examples in Emacs.  It is also related to the use of finalizers as an
alternative mechanism to stack discipline.

I am saying that there are things that could be improved:

- Emacs Lisp should provide open-directory, read-directory and
  close-directory functions.

I think that is the most important point.

There are other related observations, like:

- find-lisp-find-dired is fundamentally broken.

- the brokenness is due to directory-files being the wrong API.
  It prevents people from writing better code.

- the brokenness propagates to everything which uses directory-files,
  like find-lisp-find-dired, completion (e.g. completion on /nix/store
  blocks emacs completely for significant amount of time).  Now I see
  that even lisp/url uses directory-files thus there are cases where it
  will be unuseable.

- directory-files implies consing all the directory entries except a few
  pre-hardcoded filter use-cases.  It does not allow anything else to be
  more efficient, where one would want to filter out unneeded directory
  entries at the moment they are first encountered.

- return_count argument to directory_files_internal shows a desperate
  attempt to fix patological deficiency:-) return_count is missing from
  directory-files-recursively.  return_count still does not provide
  sufficient control of previously uncontrollable behaviour.

- directory-files-recursively brings yet another different interface for
  traversing file system

- directory-files-recursively seems to be built on different primitive
  file-name-all-completions

I do not understand the implementation of file-name-all-completions yet
but from the observed behaviour, it seems to have the same flaws like
directory-files.

> I don't see why.  A callback can easily insert files into the buffer,
> and trigger redisplay either directly or via timers and such.

But the control is in the hand of the mapping function.

Triggering redisplay is not sufficient for it to behave like in
find-dired, I think.  It needs to handle input and everything as if it
was running in a different execution thread, like find-dired does with
the "mapping function" running in an external process (find program).

> Keep in mind that when Emacs invokes an async subprocess, the display
> of what that subprocess produces is still done in the same single
> thread where the Lisp machine runs.  So we already have the
> capabilities of "pretending" we have multithreading, when these
> aspects are involved.

Yes, it works well when the mapping function (in the case of find-dired,
the find process) runs in a different execution thread/process.

It does not work well, when it runs in the same single execution thread.
See find-lisp-find-dired.

>> > There are usually protocols for the callback to stop the process and
>> > prevent the rest of objects from being examined/reported.
>> 
>> There is no such thing in directory-files.
>
> I think we have something like that in lisp/url/.

Thanks for the hint.  Unfortunatelly I can't find anything from after a
quick search.

I can see that it uses directory-files which means it also inherits its
brokenness in those cases.

>> Also, such protocols are non-trivial and usually complicate the code
>> significantly.
>
> I don't see why this would be non-trivial.  In the simplest case, the
> callback returns a special value which tells to stop the iteration.

The simplest case is also the least interesting.

> Emacs is not an OS (the popular myth about that notwithstanding).  It
> provides primitives that are useful for writing editing and
> text-processing applications.  What you don't find in Emacs is
> probably outside of its scope, or at least is rarely if ever needed.

Directory traversal seems to be essential part of Emacs.

>> But the point here was that in case of lazy sequences, the point in time
>> where close should be called is not known upfront.  One would have to
>> split the code into two parts: eager one which would force stack
>> discipline and lazy one which would do the lazy part.  Maybe doable for
>> trivial thing like lazy listing of a single directory, but it gets
>> harder when doing something non-trivial like lazy listing of a directory
>> recursively.
>
> Where you see something non-trivial, I see just simple use of our
> unwinding infrastructure.  There's no problem I see with using that
> infrastructure in recursive processing of directories.  If you see
> such problems, please tell what they are, specifically, given the
> unwind_protect functions we have and the mechanism that actually
> implements the unwinding.  In Emacs, many operations could signal an
> error or throw, and the user could type C-g at any moment, so this
> mechanism is in use all the time, and must handle correctly nested
> unwinders.

Problem is not with unwind-protect.

Problem is with directory-files and all functions that use it.  I think
I pointed out the problems sufficiently.

One fix could be implementing an alternative to directory-files,
e.g. directory mapping function, plus running it in a extra thread.

Another possibility would be to traverse the directory lazily, but that
brings an issue of eagerly closing directories.