Re: multipage html output

[Ihor Radchenko]

Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:

> To recapitulate: In my code, org-export-as calls process-multipage in
> the backend. This function:
>
> - collects and adds information necessary for org-multipage to do its
>   job, splitting the document into different parts, etc. and
>
> - then calls org-export-data on the subtrees and exports each returned
>   string to an individual file.
>
> - It finally issues a done string and executes a browser open/visit
>   file or simply exits nil.

Currently, org-export-as does the following:

1. Compute global export attributes, according to the selected export backend
2. Copy original buffer into working copy
3. Process and parse the copy, generating AST
4. Do the actual export

You plugged your multipage processing into (4), but what it actually
does involves (3), (4), and also a new kind of post-processing.
I do not think that it is a good design from the point of view of ox.el.
I prefer to reuse or extend the existing mechanisms if at all possible -
this makes new features less confusing for users and backend developers.

> - collects and adds information necessary for org-multipage to do its
>   job, splitting the document into different parts, etc. and

What you describe here is more or less what :filter-parse-tree filters
do - they can rearrange the parse tree before passing it to the
transcoders. Why not reusing it for multipage export?

> - then calls org-export-data on the subtrees and exports each returned
>   string to an individual file.

And you simply call `org-export-transcode-page' for this, followed by
writing the returned string to file.

The first part can fit within `org-export-as', but writing to file is
going a step beyond, duplicating what `org-export-to-file' does.

> - It finally issues a done string and executes a browser open/visit
>   file or simply exits nil.

... which again steps beyond `org-export-as' scope - post-processing is
currently done as a part of `org-export-to-file'/`org-export-to-buffer'.

----

Let me propose the following changes to ox.el:

1. org-data will be transcoded using `org-export-transcode-org-data',
   which can be overridden by setting org-data transcoders in the
   individual backends.

2. org-export-as will understand transcoded output to be a list of
   strings and will transfer INFO plist as text property in the return
   values

3. org-export-to-file will make use of the text properties to retrieve
   the file name to write.  This way, export backend itself can assign
   the file names where each exporter string should go.

I believe that my changes should allow you to implement multipage export
in the following way:

1. You can use :filter-parse-tree in ox-html backend to replace the
   original (org-data ...) AST with a list of
   ((org-page ...) (org-page ...) ...) pseudo-elements and populate INFO
   channel with auxiliary information you now compute in 
`org-html-process-multipage'

2. You can define org-page transcoder to render individual pages as
needed

3. You can assign :output-file text property to the returned org-page
   strings and use org-export-to-file to generate the multipage output
   on disk

4. You can handle opening exported files by augmenting POST-PROCESS
   argument in `org-html-export-to-multipage-html' and calling
   `org-export-file' instead of `org-export-as'.

The tentative patches (against Org mode main branch) implementing my
changes are attached.

>From 540c8ef21c26df79cf48f58afb4e88130985e2f7 Mon Sep 17 00:00:00 2001
Message-ID: 
<540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 24 Jul 2024 11:40:57 +0200
Subject: [PATCH 1/3] ox: Factor out org-data transcoding into dedicated
 overrideable transcoder

* lisp/ox.el (org-export-transcode-org-data): New function serving as
the default transcoder for org-data export.
(org-export-transcoder): Use `org-export-transcode-org-data' when no
org-data transcoder is defined.
(org-export-as): Rely upon org-data transcoder to do its job.
---
 lisp/ox.el | 55 +++++++++++++++++++++++++++++-------------------------
 1 file changed, 30 insertions(+), 25 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index fbd9bb0df..bdee71082 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1883,9 +1883,11 @@ (defun org-export-transcoder (blob info)
 INFO is a plist containing export directives."
   (let ((type (org-element-type blob)))
     ;; Return contents only for complete parse trees.
-    (if (eq type 'org-data) (lambda (_datum contents _info) contents)
-      (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
-       (and (functionp transcoder) transcoder)))))
+    (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
+      (cond
+       ((functionp transcoder) transcoder)
+       ;; Use default org-data transcoder unless specified.
+       ((eq type 'org-data) #'org-export-transcode-org-data)))))
 
 (defun org-export--keep-spaces (data info)
   "Non-nil, when post-blank spaces after removing DATA should be preserved.
@@ -3004,31 +3006,34 @@ (defun org-export-as
                        backend info subtreep visible-only ext-plist))
           ;; Eventually transcode TREE.  Wrap the resulting string into
           ;; a template.
-          (let* ((body (org-element-normalize-string
-                        (or (org-export-data (plist-get info :parse-tree) info)
-                             "")))
-                 (inner-template (cdr (assq 'inner-template
-                                            (plist-get info 
:translate-alist))))
-                 (full-body (org-export-filter-apply-functions
-                             (plist-get info :filter-body)
-                             (if (not (functionp inner-template)) body
-                               (funcall inner-template body info))
-                             info))
-                 (template (cdr (assq 'template
-                                      (plist-get info :translate-alist))))
-                  (output
-                   (if (or (not (functionp template)) body-only) full-body
-                    (funcall template full-body info))))
+          (let ((output
+                  (or (org-export-data (plist-get info :parse-tree) info)
+                      "")))
              ;; Call citation export finalizer.
              (when (plist-get info :with-cite-processors)
                (setq output (org-cite-finalize-export output info)))
-            ;; Remove all text properties since they cannot be
-            ;; retrieved from an external process.  Finally call
-            ;; final-output filter and return result.
-            (org-no-properties
-             (org-export-filter-apply-functions
-              (plist-get info :filter-final-output)
-              output info)))))))))
+             (let ((filters (plist-get info :filter-final-output)))
+               ;; Remove all text properties since they cannot be
+              ;; retrieved from an external process.  Finally call
+              ;; final-output filter and return result.
+               (org-no-properties
+                (org-export-filter-apply-functions filters output 
info))))))))))
+
+(defun org-export-transcode-org-data (_ body info)
+  "Transcode `org-data' node with BODY.  Return transcoded string.
+INFO is the communication channel plist."
+  (let* ((inner-template (cdr (assq 'inner-template
+                                   (plist-get info :translate-alist))))
+        (full-body (org-export-filter-apply-functions
+                    (plist-get info :filter-body)
+                    (if (not (functionp inner-template)) body
+                      (funcall inner-template body info))
+                    info))
+        (template (cdr (assq 'template
+                             (plist-get info :translate-alist))))
+         (body-only (memq 'body-only (plist-get info :export-options))))
+    (if (or (not (functionp template)) body-only) full-body
+      (funcall template full-body info))))
 
 (defun org-export--annotate-info (backend info &optional subtreep visible-only 
ext-plist)
   "Annotate the INFO plist according to the BACKEND.
-- 
2.45.2

>From 1b0b331f92abc1ca7e04f71fe7ff60da57c719b8 Mon Sep 17 00:00:00 2001
Message-ID: 
<1b0b331f92abc1ca7e04f71fe7ff60da57c719b8.1721815865.git.yantar92@posteo.net>
In-Reply-To: 
<540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
References: 
<540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 24 Jul 2024 11:51:21 +0200
Subject: [PATCH 2/3] org-export-as: Allow the return value to be a list of
 strings; add INFO

* lisp/ox.el (org-export-as): Allow the transcoders to return list of
strings and return it.  When returning a string, put INFO plist as
text property.  Do not remove text properties assigned by the
transcoders.
(org-export-data): Document that list of strings may be returned.
---
 lisp/ox.el | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index bdee71082..a76b3b353 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -1930,7 +1930,7 @@ (defun org-export-data (data info)
 
 The `:filter-parse-tree' filters are not applied.
 
-Return a string."
+Return a string or a list of strings."
   (or (gethash data (plist-get info :exported-data))
       ;; Handle broken links according to
       ;; `org-export-with-broken-links'.
@@ -2969,7 +2969,9 @@ (defun org-export-as
 with external parameters overriding Org default settings, but
 still inferior to file-local settings.
 
-Return code as a string."
+Return code as a string or a list of strings.
+The returned strings will have their `org-export-info' property set to
+export information channel."
   (when (symbolp backend) (setq backend (org-export-get-backend backend)))
   (org-export-barf-if-invalid-backend backend)
   (org-fold-core-ignore-modifications
@@ -3009,15 +3011,25 @@ (defun org-export-as
           (let ((output
                   (or (org-export-data (plist-get info :parse-tree) info)
                       "")))
+             (setq output (ensure-list output))
              ;; Call citation export finalizer.
              (when (plist-get info :with-cite-processors)
-               (setq output (org-cite-finalize-export output info)))
+               (setq output
+                     (mapcar
+                      (lambda (o) (org-cite-finalize-export o info))
+                      output)))
              (let ((filters (plist-get info :filter-final-output)))
-               ;; Remove all text properties since they cannot be
-              ;; retrieved from an external process.  Finally call
-              ;; final-output filter and return result.
-               (org-no-properties
-                (org-export-filter-apply-functions filters output 
info))))))))))
+               ;; Call final-output filter and return result.
+               (setq output
+                     (mapcar
+                      (lambda (o) (org-export-filter-apply-functions filters o 
info))
+                      output)))
+             ;; Apply org-export-info property.
+             (setq output
+                   (mapcar
+                    (lambda (o) (org-add-props o nil 'org-export-info info))
+                    output))
+             (if (length= output 1) (car output) output))))))))
 
 (defun org-export-transcode-org-data (_ body info)
   "Transcode `org-data' node with BODY.  Return transcoded string.
-- 
2.45.2

>From 6fa2efadd229a667fba1b18aecc9d1ead5f284ac Mon Sep 17 00:00:00 2001
Message-ID: 
<6fa2efadd229a667fba1b18aecc9d1ead5f284ac.1721815865.git.yantar92@posteo.net>
In-Reply-To: 
<540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
References: 
<540c8ef21c26df79cf48f58afb4e88130985e2f7.1721815865.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Wed, 24 Jul 2024 12:09:36 +0200
Subject: [PATCH 3/3] org-export-to-file: Derive file name to write from export
 output

* lisp/ox.el (org-export--write-output): New helper function
performing writing an export output or a list of outputs to file.  It
derives the file name from :output-file property in the output string
or INFO plist stored in the output string.
(org-export-to-file): Handle export output being a list of strings.
Use `org-export--write-output'.
---
 lisp/ox.el | 61 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 23 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index a76b3b353..d78c04998 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -6830,6 +6830,31 @@   (defun org-latex-export-as-latex
        (switch-to-buffer-other-window buffer))
       buffer)))
 
+(defun org-export--write-output (output encoding)
+  "Write OUTPUT to file with ENCODING.
+OUTPUT may be a string or a list of strings.
+The target file is retrieved from :output-file OUTPUT property or
+:output-file property in plist stored in `org-export-info' property of
+each string.
+
+Return the file name or a list of file names."
+  (if (listp output) (mapcar #'org-export--write-output output)
+    (let ((file (or
+                 (get-text-property 0 :output-file output)
+                 (plist-get
+                  (get-text-property 0 'org-export-info output)
+                  :output-file))))
+      (with-temp-buffer
+        (insert output)
+        ;; Ensure final newline.  This is what was done
+        ;; historically, when we used `write-file'.
+        ;; Note that adding a newline is only safe for
+        ;; non-binary data.
+        (unless (bolp) (insert "\n"))
+        (let ((coding-system-for-write encoding))
+         (write-region nil nil file))
+        file))))
+
 ;;;###autoload
 (defun org-export-to-file
     (backend file &optional async subtreep visible-only body-only ext-plist
@@ -6878,33 +6903,23 @@   (defun org-latex-export-to-latex
            `(let ((output
                    (org-export-as
                     ',backend ,subtreep ,visible-only ,body-only
-                    ',ext-plist)))
-              (with-temp-buffer
-                (insert output)
-                 ;; Ensure final newline.  This is what was done
-                 ;; historically, when we used `write-file'.
-                 ;; Note that adding a newline is only safe for
-                 ;; non-binary data.
-                 (unless (bolp) (insert "\n"))
-                (let ((coding-system-for-write ',encoding))
-                  (write-region nil nil ,file)))
-              (or (ignore-errors (funcall ',post-process ,file)) ,file)))
+                    ',ext-plist))
+                   file)
+               (setq file (org-export--write-output output ',encoding))
+               (let ((post (lambda (f) (or (ignore-errors (funcall 
',post-process f)) f))))
+                 (if (listp file) (mapcar post file) (funcall post file)))))
         (let ((output (org-export-as
-                       backend subtreep visible-only body-only ext-plist)))
-          (with-temp-buffer
-            (insert output)
-            ;; Ensure final newline.  This is what was done
-            ;; historically, when we used `write-file'.
-            ;; Note that adding a newline is only safe for
-            ;; non-binary data.
-            (unless (bolp) (insert "\n"))
-            (let ((coding-system-for-write encoding))
-             (write-region nil nil file)))
+                       backend subtreep visible-only body-only ext-plist))
+              file)
+          (setq file (org-export--write-output output encoding))
           (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p 
output))
             (org-kill-new output))
           ;; Get proper return value.
-          (or (and (functionp post-process) (funcall post-process file))
-             file))))))
+          (let ((post (lambda (f)
+                        (or (and (functionp post-process)
+                                 (funcall post-process f))
+                           f))))
+            (if (listp file) (mapcar post file) (funcall post file))))))))
 
 (defun org-export-output-file-name (extension &optional subtreep pub-dir)
   "Return output file's name according to buffer specifications.
-- 
2.45.2

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>