bug#68559: [PATCH] Improve Python shell completion

[kobarity]

Mattias Engdegård wrote:
> 21 feb. 2024 kl. 14.13 skrev kobarity <kobarity@gmail.com>:
> > As far as I have tried, native completions cannot be enabled for
> > libedit-based readline package, even on Linux.
> Did you find out why? Python responds nicely to TAB if run from a terminal, 
> so what is it that we do in Emacs that prevents it from doing so? A TTY 
> setting, or an environment variable (the TERM=dumb)?

No, not exactly, but I don't think it is related to terminal settings.
python.el does not parse the completion candidates shown on the
terminal.  Instead, it expects the candidates in a particular format,
which I call the protocol between python.el and inferior Python
process.

__PYTHON_EL_native_completion_setup() defined in
`python-shell-completion-native-setup' sets up the completer to do so.
We can test it by removing empty lines and pasting into Python REPL
outside Emacs.  An example of typing "ra" and TAB using readline would
look like this.

>>> raraise
range

0__dummy_completion__  1__dummy_completion__
>>> ra

However, when I use libedit-based readline and remove the raise line
in __PYTHON_EL_native_completion_setup(), it behaves like this.

>>> raraise
range

Please note that there is no dummy completions, and the new prompt is
not shown.  When I hit TAB again, it would be like this.

>>> raraise
range
      raise
range

0__dummy_completion__ 0__dummy_completion__ 0__dummy_completion__ 
1__dummy_completion__
>>> ra

I think this difference in behavior is the reason why Native
completions does not work with libedit.

> > So, there are two ways for Mac users to avoid the warning.  One is to
> > use gnureadline instead of libedit, and the other is to give up native
> > completions and set `python-shell-completion-native-enable' to nil.
> 
> Preferably neither. Emacs should adapt to the system environment, and in 
> particular not warn about the default environment. A warning is an indication 
> of a possible risk or other problem, and there is none here.
> 
> At most, python-mode could show a calm notice on the echo line but even that 
> is a stretch. What do you think about this rough patch?

Personally, I think this patch would be fine.

I think we can also improve Non-native completions.  Current
implementation sends the definition of __PYTHON_EL_get_completions()
every time.  However, sending it once during initialization should be
sufficient.  Worse, if the number of characters sent exceeds
`comint-max-line-length', it will be sent via file.  This is happening
in your environment.  Here is the log you presented.

Test python-completion-at-point-1 backtrace:
  json-parse-string("__PYTHON_EL_eval_file(\"/var/folders/qy/zstv16390

Thanks to the echo back, we can see __PYTHON_EL_eval_file() was used.

> > I think the protocol between python.el and inferior Python process is
> > already platform independent.  Protocol violations are echo back.
> 
> No, I meant a protocol that allows Emacs to act as a first-class Python 
> front-end, not simulate a terminal, send keystrokes, use heuristics to 
> determine what in the output stream is a prompt, REPL value, error, 
> completion etc.
> 
> For example, it's a bit silly to input multi-line code in the REPL as a 
> sequence of individual single-line commands, when we actually are inside a 
> text editor that can edit multi-line Python code without a problem.
> 
> (I'm not suggesting that you or anybody in particular should do this; just 
> that it's feasible. It would clearly be quite some work!)

I am not opposed to this approach, but as you wrote, it is very
different from the current inferior Python implementation.
Jupyter-emacs's approach may be similar to some extent.  It uses zmq
as the communication channel.

https://github.com/emacs-jupyter/jupyter

> >> Thanks for your patches. I suggest we apply your set-tty-raw patch on 
> >> master now since it cures the test failures without breaking anything else 
> >> (on Mac; I'm assuming no regression elsewhere).
> >> Would you like me to do that for you?
> > Yes, please.
> Done!

Thank you!