Bug: Double trailing slash for default candidate in org-refile-get-target [9.4 (9.4-7-g3eccc5-elpaplus @ /home/gustavo/.emacs.d/elpa/org-plus-contrib-20200921/)]

[Gustavo Barros]

Hi All,

some time ago, I've reported an issue regarding duplicity of the default 
candidate in `org-refile' 
(https://orgmode.org/list/87lftw1k2n.fsf@gmail.com/).  The problem was 
that, when using `org-refile-use-outline-path' an "extra" slash was 
appended at the end of every path, but candidates were stored in 
`org-refile-history' without that extra slash.  Bastien took care of 
that and indeed changed things so as to pass the elements to 
`org-refile-history' with the trailing slash as appropriate.
At the time, I reported a difference of behavior between 
`completing-read-default' and `ivy-completing-read' after the mentioned 
commit by Bastien.  But the issue did not appear for Bastien, which does 
not use Ivy, and I also was not able to diagnose the problem properly. 
I felt I didn't have enough to offer as to insist, so I resorted to an 
old hack of mine.  But the new release this week (thank you very much!, 
btw) has bitten me again on this, so I went back to some digging, and 
hopefully I can do a better job this time in diagnosing the problem and 
suggesting a fix.

An ECM to reproduce the issue as it currently stands is:

- Start 'emacs -Q'

- Do an initial setup:
 #+begin_src emacs-lisp
 (add-to-list 'load-path "~/.emacs.d/elpa/org-plus-contrib-20200921")
 (add-to-list 'load-path "~/.emacs.d/elpa/ivy-20200826.955")
 ;; Those are the latest Org weekly build (Org 9.4) and the current up 
 to date
 ;; Ivy at Melpa
 (setq org-refile-targets '(("~/org/test.org" :maxlevel . 2)))
 (setq org-refile-use-outline-path 'file)
 (setq org-outline-path-complete-in-steps nil)
 (require 'ivy)
 (ivy-mode)
 ;; Bear with me, the problem is not with Ivy, I'll demonstrate that.
 #+end_src

- Open file "~/org/test.org", with contents:
 #+begin_src org
 ,* Top heading 1
 ,* Top heading 2
 ,** Entry 1
 ,** Entry 2
 #+end_src

- Go to heading "Entry 1", refile it to "Top heading 1"

- Go to heading "Entry 2", and call `org-refile'

- Observe the available candidates, and notice "test.org/Top heading 1/" 
 is there twice, once as the default candidate, with a *double* 
 trailing slash, and also on the paths list, with a single trailing 
 slash.

I've tried to pin down what is going on here and my understanding is 
that Bastien's fix on that previous thread did indeed correct the 
problem of the missing trailing slash in `org-refile-history' and this 
indeed corresponds correctly to the state of the completion "collection" 
(the let bound `tbl' variable in `org-refile-get-location'), as it 
should.  But there remained a couple of instances in 
`org-refile-get-location' which added the trailing slash considering 
`org-refile-history' didn't have them, so that when this is done, we get 
a double trailing slash.
The two instances are: 1) when the completion function is actually 
called:
#+begin_src emacs-lisp
(setq answ (funcall cfunc prompt tbl nil (not new-nodes)
                            nil 'org-refile-history
			    (or cdef (concat (car org-refile-history) 
			    extra))))
#+end_src

2) And three lines bellow, on the let binding:

#+begin_src emacs-lisp
(let* ((last-refile-loc (car org-refile-history))
      (last-refile-loc-path (concat last-refile-loc extra)))
 ...)
#+end_src

In both instances, we are getting the `car' of `org-refile-history' 
which now already has `extra' (that is, the trailing slash) and adding 
it again.
My suggested fix is to remove these `extra's in duplicity, they are 
remnants of when `org-refile-history' didn't have them already.  That 
is:
#+begin_src emacs-lisp
(setq answ (funcall cfunc prompt tbl nil (not new-nodes)
                            nil 'org-refile-history
                            (or cdef (car org-refile-history))))
#+end_src

And:

#+begin_src emacs-lisp
(let* ((last-refile-loc (car org-refile-history))
      (last-refile-loc-path last-refile-loc))
 ...)
#+end_src

Of course, the second opens some opportunity to simplify the code that 
follows considering `last-refile-loc-path' and `last-refile-loc' are now 
identical.

Why I think this is the problem and the correct way to fix it:

1) If you add inspection points at the appropriate locations for the 
sexps =(concat (car org-refile-history) extra)= and =(concat 
last-refile-loc extra)= you will find the double trailing slash there, 
and it shouldn't be there.
2) The visual manifestation of this double trailing slash in the default 
candidate with `ivy-mode' is there therefore independently of 
`ivy-mode`.  Indeed, `ivy-mode' basically sets 
`completing-read-function' to `ivy-completing-read', which in turn calls 
its main API function `ivy-read'.  `ivy-completing-read' just passes 
along the `def' argument without manipulating it.  As far as I can see, 
`ivy-read' also does not change it before calling 
`read-from-minibuffer'.
Why `completing-read-default' does not show this second trailing slash 
of the `def' argument it received is another matter.  But it's there...
3) I've tested this suggested fix in the following scenarios (starting 
from the ECM above):
#+begin_src emacs-lisp
(setq org-refile-use-outline-path 'file)
(setq org-outline-path-complete-in-steps nil)
;; (ivy-mode)
#+end_src

#+begin_src emacs-lisp
(setq org-refile-use-outline-path 'file)
(setq org-outline-path-complete-in-steps nil)
(ivy-mode)
#+end_src

#+begin_src emacs-lisp
(setq org-refile-use-outline-path 'file)
(setq org-outline-path-complete-in-steps t)
;; `org-olpath-completing-read' does not play well with `ivy-mode' so no 
  need
;; to use it here
#+end_src

#+begin_src emacs-lisp
(setq org-refile-use-outline-path nil)
(ivy-mode)
#+end_src

#+begin_src emacs-lisp
(setq org-refile-use-outline-path nil)
;; (ivy-mode)
#+end_src

Other non-nil values of `org-refile-use-outline-path' should be 
equivalent to `file' with respect to the issue at hand, so they are 
implicitly covered.
Having tested these cases, to the best of my knowledge, they all work as 
expected.  Though I'm admittedly less acquainted with =(setq 
org-refile-use-outline-path nil)= and =(setq 
org-outline-path-complete-in-steps t)=.
All tests and ECM were ran with "Org mode version 9.4 
(9.4-7-g3eccc5-elpaplus @ 
/home/gustavo/.emacs.d/elpa/org-plus-contrib-20200921/)" and "GNU Emacs 
27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 
1.16.0) of 2020-08-11".
I hope this is a more useful report than last time.


Best,
Gustavo.





Emacs  : GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 
3.24.20, cairo version 1.16.0)
of 2020-08-11
Package: Org mode version 9.4 (9.4-7-g3eccc5-elpaplus @ 
/home/gustavo/.emacs.d/elpa/org-plus-contrib-20200921/)
current state:
==============
(setq
org-src-mode-hook '(org-src-babel-configure-edit-buffer
                     org-src-mode-configure-edit-buffer)
org-link-shell-confirm-function 'yes-or-no-p
org-metadown-hook '(org-babel-pop-to-session-maybe)
org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
org-refile-targets '(("~/org/test.org" :maxlevel . 2))
org-mode-hook '(#[0 "\300\301\302\303\304$\207"
		   [add-hook change-major-mode-hook org-show-all append 
		   local]
                   5]
                 #[0 "\300\301\302\303\304$\207"
		   [add-hook change-major-mode-hook 
		   org-babel-show-result-all
                    append local]
                   5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes
                 org-eldoc-load)
org-outline-path-complete-in-steps nil
org-archive-hook '(org-attach-archive-delete-maybe)
org-confirm-elisp-link-function 'yes-or-no-p
org-agenda-before-write-hook '(org-agenda-add-entry-text)
org-metaup-hook '(org-babel-load-in-session-maybe)
org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 
"\n\n(fn ENTRY)"]
org-babel-pre-tangle-hook '(save-buffer)
org-tab-first-hook '(org-babel-hide-result-toggle-maybe
                      org-babel-header-arg-expand)
org-agenda-loop-over-headlines-in-active-region nil
org-src-lang-modes '(("arduino" . arduino) ("redis" . redis) ("php" 
. php)
                      ("C" . c) ("C++" . c++) ("asymptote" . asy)
		      ("bash" . sh) ("beamer" . latex) ("calc" 
		      . fundamental)
		      ("cpp" . c++) ("ditaa" . artist) ("dot" 
		      . fundamental)
                      ("elisp" . emacs-lisp) ("ocaml" . tuareg)
                      ("screen" . shell-script) ("shell" . sh)
                      ("sqlite" . sql))
org-occur-hook '(org-first-headline-recenter)
org-cycle-hook '(org-cycle-hide-archived-subtrees 
org-cycle-hide-drawers
                  org-cycle-show-empty-lines
                  org-optimize-window-after-visibility-change)
org-speed-command-hook '(org-speed-command-activate
                          org-babel-speed-command-activate)
org-refile-use-outline-path 'file
org-export-before-parsing-hook '(org-attach-expand-links)
org-confirm-shell-link-function 'yes-or-no-p
org-link-parameters '(("attachment" :follow org-attach-follow :complete
                        org-attach-complete-link)
                       ("id" :follow org-id-open)
		       ("eww" :follow org-eww-open :store 
		       org-eww-store-link)
                       ("rmail" :follow org-rmail-open :store
                        org-rmail-store-link)
		       ("mhe" :follow org-mhe-open :store 
		       org-mhe-store-link)
		       ("irc" :follow org-irc-visit :store 
		       org-irc-store-link
                        :export org-irc-export)
		       ("info" :follow org-info-open :export 
		       org-info-export
                        :store org-info-store-link)
                       ("gnus" :follow org-gnus-open :store
                        org-gnus-store-link)
                       ("docview" :follow org-docview-open :export
			org-docview-export :store 
			org-docview-store-link)
                       ("bibtex" :follow org-bibtex-open :store
                        org-bibtex-store-link)
		       ("bbdb" :follow org-bbdb-open :export 
		       org-bbdb-export
                        :complete org-bbdb-complete-link :store
                        org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link) ("file+sys")
		       ("file+emacs") ("shell" :follow 
		       org-link--open-shell)
                       ("news" :follow
                        #[514 "\301\300\302Q\"\207"
                          ["news" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("mailto" :follow
                        #[514 "\301\300\302Q\"\207"
			  ["mailto" browse-url ":"] 6 "\n\n(fn URL 
			  ARG)"]
                        )
                       ("https" :follow
                        #[514 "\301\300\302Q\"\207"
                          ["https" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("http" :follow
                        #[514 "\301\300\302Q\"\207"
                          ["http" browse-url ":"] 6 "\n\n(fn URL ARG)"]
                        )
                       ("ftp" :follow
			#[514 "\301\300\302Q\"\207" ["ftp" browse-url 
                        ":"]
                          6 "\n\n(fn URL ARG)"]
                        )
                       ("help" :follow org-link--open-help)
                       ("file" :complete org-link-complete-file)
                       ("elisp" :follow org-link--open-elisp)
                       ("doi" :follow org-link--open-doi))
org-link-elisp-confirm-function 'yes-or-no-p
)