Re: [PATCH] Async evaluation in ob-shell

[Matt]

 > Matt matt@excalamus.com> writes:
 >
 > I see only two options to fix it: remove a space from the concat expression 
 > (which I did in my latest patch) or remove a space from 
 > `org-babel-sh-prompt'.

Unfortunately, I was mistaken and the second option (removing the space from 
`org-babel-sh-prompt') doesn't fix the issue.  The TLDR is that the code in 
`org-babel-comint-async-filter' which grabs the region between the indicators 
(incorrectly) fails to include the prompt's trailing space.

#+begin_longwinded_explanation
I'll first explain why removing the space from `org-babel-sh-prompt' doesn't 
fix the issue because it well also highlight the underlying problem.

If we remove the space from the `org-babel-sh-prompt', then 
`comint-prompt-regexp' becomes "^org_babel_sh_prompt> *" (with one space).   
This would work if the string passed to the `ob-shell-async-chunk-callback' 
stayed the same.  It doesn't (this is where my reasoning and testing failed).  
Changing the `org-babel-sh-prompt' to "org_babel_sh_prompt>" (without a space) 
causes the following string to be passed to the callback:

"org_babel_sh_prompt>1
org_babel_sh_prompt>2
org_babel_sh_prompt"

Note that the final prompt doesn't have a ">" and therefore the 
`comint-prompt-regexp' (which becomes "^org_babel_sh_prompt> * (with one 
space)) used in the callback fails to match it.  When we remove the space from 
the `org-babel-sh-prompt', the session buffer looks like this:

"sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt>";PS2=
org_babel_sh_prompt>echo 
'ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8'
echo 1
echo 2
echo 'ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8'
ob_comint_async_shell_start_39610981-1020-4baf-9dfb-f96d10af1cf8
org_babel_sh_prompt>1
org_babel_sh_prompt>2
org_babel_sh_prompt>ob_comint_async_shell_end_39610981-1020-4baf-9dfb-f96d10af1cf8
org_babel_sh_prompt>"

The `org-babel-comint-async-filter' is what calls the 
`ob-shell-async-chunk-callback' (ob-comint.el:284).  It monitors for the end 
indicator.  When that appears, it passes the region between the beginning of 
the end indicator **less 1** and the character after the end of the start 
indicator to the callback.  For a clean run of 
`test-ob-shell/session-async-evaluation', the beginning of the end indicator is 
at 361 and the character after the end of the start indicator is at 298.  This 
is the string I gave above which is missing the ">".  

In order to make the second option work, we'd need to change the "less 1" part 
of `org-babel-comint-async-filter' from (- (match-beginning 0) 1) to 
(match-beginning 0).   It turns out that's actually all we need to do.

When `org-babel-sh-prompt' is "org_babel_sh_prompt> " (with one space), then 
the session buffer looks like:

"sh-5.1$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> echo 
'ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26'
echo 1
echo 2
echo 'ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26'
ob_comint_async_shell_start_3270ed43-a99b-423f-a5fa-b15fb2e4ae26
org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt> 
ob_comint_async_shell_end_3270ed43-a99b-423f-a5fa-b15fb2e4ae26
org_babel_sh_prompt> "

The region passed to the callback is then defined as 366 to 300, or

"org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt>"  (<-- no space)

This looks okay at first glance.  However, **the last line is not a valid 
prompt**.  A prompt must end in a space!  When the `org-babel-sh-prompt' is set 
to  "org_babel_sh_prompt> " (with one space), the `comint-prompt-regexp' is 
"^org_babel_sh_prompt>  *" (with two spaces).  This means that the 
`comint-prompt-regexp' matches on a trailing space which the **region passed to 
the callback doesn't have**.  Therefore, the match fails.

Instead, if we modify the `org-babel-comint-async-filter' like

modified   lisp/ob-comint.el
@@ -273,7 +273,7 @@ STRING contains the output originally inserted into the 
comint buffer."
                   (res-str-raw
                    (buffer-substring
                     ;; move point to beginning of indicator
-                     (- (match-beginning 0) 1)
+                     (match-beginning 0)
                     ;; find the matching start indicator
                     (cl-loop
                       do (re-search-backward indicator)

then the region passed to the callback will be from 367 to 300, or

"org_babel_sh_prompt> 1
org_babel_sh_prompt> 2
org_babel_sh_prompt> " (<-- with one space)

The `comint-prompt-regexp' will now match the last prompt in the region.

With this change, the `org-babel-sh-prompt' keeps the trailing space (like it 
should), the `comint-prompt-regexp' becomes "^org_babel_sh_prompt>  *" (with 
two spaces, requiring a prompt to have a trailing space like it should), the 
`ob-shell-async-chunk-callback' can use `comint-prompt-regexp' without 
modification, and the tests all pass.
#+end_longwinded_explanation

I've attached an updated diff.  If everyone is satisfied with this, I'll do a 
proper commit and then handle moving the uuid code like we talked about earlier 
in the thread.

0004-ob-shell-Add-async-evaluation.diff
Description: Binary data