bug#24189: 25.1.50; args-out-of-range in ansi-color-filter-apply

[Aaron Ecay]

With recent master emacs, I get an error under the following conditions:

0. Install ipython
1. emacs -Q
2. Eval in scratch buffer: (setq python-shell-interpreter "ipython")
3. C-x C-f foo.py
4. C-c C-p

The backtrace is:

-----cut here-----

Debugger entered--Lisp error: (args-out-of-range "Python 3.5.2 (default, Jun 28 
2016, 08:46:01) 
Type \"copyright\", \"credits\" or \"license\" for more information.

IPython 5.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

[6n[0m[0m[J[J[0m[6n" 423)
  string-match("" "Python 3.5.2 (default, Jun 28 2016, 08:46:01) \nType 
\"copyright\", \"credits\" or \"license\" for more information.\n\nIPython 
5.0.0 -- An enhanced Interactive Python.\n?         -> Introduction and 
overview of IPython's features.\n%quickref -> Quick reference.\nhelp      -> 
Python's own help system.\nobject?   -> Details about 'object', use 'object??' 
for extra details.\n\n[6n[0m[0m[J[J[0m[6n" 423)
  ansi-color-filter-apply("Python 3.5.2 (default, Jun 28 2016, 08:46:01) \nType 
\"copyright\", \"credits\" or \"license\" for more information.\n\nIPython 
5.0.0 -- An enhanced Interactive Python.\n?         -> Introduction and 
overview of IPython's features.\n%quickref -> Quick reference.\nhelp      -> 
Python's own help system.\nobject?   -> Details about 'object', use 'object??' 
for extra details.\n\n[?1l[6n[?2004h[?25l[?7l[0m[0m[J[?12l[?25h[J[0m[?2004l[6n")
  python-shell-comint-watch-for-first-prompt-output-filter("Python 3.5.2 
(default, Jun 28 2016, 08:46:01) \nType \"copyright\", \"credits\" or 
\"license\" for more information.\n\nIPython 5.0.0 -- An enhanced Interactive 
Python.\n?         -> Introduction and overview of IPython's 
features.\n%quickref -> Quick reference.\nhelp      -> Python's own help 
system.\nobject?   -> Details about 'object', use 'object??' for extra 
details.\n\n[?1l[6n[?2004h[?25l[?7l[0m[0m[J[?12l[?25h[J[0m[?2004l[6n")
  run-hook-with-args(python-shell-comint-watch-for-first-prompt-output-filter 
"Python 3.5.2 (default, Jun 28 2016, 08:46:01) \nType \"copyright\", 
\"credits\" or \"license\" for more information.\n\nIPython 5.0.0 -- An 
enhanced Interactive Python.\n?         -> Introduction and overview of 
IPython's features.\n%quickref -> Quick reference.\nhelp      -> Python's own 
help system.\nobject?   -> Details about 'object', use 'object??' for extra 
details.\n\n[?1l[6n[?2004h[?25l[?7l[0m[0m[J[?12l[?25h[J[0m[?2004l[6n")
  comint-output-filter(#<process Python> "Python 3.5.2 (default, Jun 28 2016, 
08:46:01) \nType \"copyright\", \"credits\" or \"license\" for more 
information.\n\nIPython 5.0.0 -- An enhanced Interactive Python.\n?         -> 
Introduction and overview of IPython's features.\n%quickref -> Quick 
reference.\nhelp      -> Python's own help system.\nobject?   -> Details about 
'object', use 'object??' for extra 
details.\n\n[?1l[6n[?2004h[?25l[?7l[0m[0m[J[?12l[?25h[J[0m[?2004l[6n")

-----cut here-----

The relevant code is:

(defun ansi-color-filter-apply (string)
  "Filter out all ANSI control sequences from STRING.

Every call to this function will set and use the buffer-local variable
`ansi-color-context' to save partial escape sequences.  This information
will be used for the next call to `ansi-color-apply'.  Set
`ansi-color-context' to nil if you don't want this.

This function can be added to `comint-preoutput-filter-functions'."
  (let ((start 0) end result)
    ;; if context was saved and is a string, prepend it
    (if (cadr ansi-color-context)
        (setq string (concat (cadr ansi-color-context) string)
              ansi-color-context nil))
    ;; find the next escape sequence
    (while (setq end (string-match ansi-color-regexp string start))
      (setq result (concat result (substring string start end))
            start (match-end 0)))
    ;; eliminate unrecognized escape sequences
    (while (string-match ansi-color-drop-regexp string)
      (setq string
            (replace-match "" nil nil string)))
    ;; save context, add the remainder of the string to the result
    (let (fragment)
      (if (string-match "\033" string start) ;; <---- ?!?!?!?
          (let ((pos (match-beginning 0)))
            (setq fragment (substring string pos)
                  result (concat result (substring string start pos))))
        (setq result (concat result (substring string start))))
      (setq ansi-color-context (if fragment (list nil fragment))))
    result))

It looks obviously bogus: the value of start is computed in the first
while, but the second while (potentially) makes deletions in the string.
After the second while, the value of start is reused (on the line I
marked with a comment), when it could be (and in this case is) beyond
the end of the string because of the deletions.  I don’t understand the
code well enough to know what the proper fix is.  I tried moving the
“eliminate unrecognized escape sequences” while to before the “find the
next escape sequence” one.  That made the error disappear, but did not
successfully display ANSI colors in the buffer (but rather left many
escape sequences in raw form behind).

Thanks,
Aaron


In GNU Emacs 25.1.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.20.8)
 of 2016-08-08 built on haize
Repository revision: 9fc22fb932599fe4fecffffa920abe509ab5cbb0
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description:     Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man
 --pdfdir=/usr/share/doc/emacs/pdf --with-sound=alsa --without-gconf
 --with-x-toolkit=gtk3 --with-xft 'CFLAGS=-march=x86-64 -mtune=generic
 -O2 -pipe -fstack-protector --param=ssp-buffer-size=4'
 CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

-- 
Aaron Ecay