bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Eli Zaretskii]

> From: Ivan Andrus <darthandrus@gmail.com>
> Date: Sun, 13 Apr 2014 19:43:08 -0600
> Cc: 17244-done@debbugs.gnu.org
> 
> > <rant>
> > To tell the truth, Lisp code which covers large portions of buffer
> > text with much shorter display strings that include newlines deserves
> > to be broken.  The current Emacs display engine was never designed to
> > handle situations where the displayed text is so starkly different
> > from buffer text, so the result of trying to fix "bugs" such as this
> > one is a never-ending series of band-aids, one upon the other, which
> > make the code utterly incomprehensible and unmaintainable.
> > 
> > So I'm this close to refusing to fix such "bugs", and instead asking
> > the authors of such Lisp to either find more benign ways of expressing
> > what they need, or work around the limitations of the display engine
> > in their own Lisp.
> > </rant>     
> 
> So is the problem having newlines in the 'display property, or hiding large 
> portions of the buffer?

Each one of these is a problem.  When they are present together in the
same display string, the problems grow exponentially.

> Because, in my original use case (my modified version of 
> http://www.emacswiki.org/emacs/fold.el) I don’t think I have newlines in the 
> display property, though I definitely hide large portions of the buffer.

This is enough to cause problems.  Consider what Emacs needs to do
when you invoke "C-u N C-p" or "(line-move-visual -N)".  It needs to
find out what is the buffer position N screen lines before point.  But
where in the buffer is that?  When those N lines display what is
mostly buffer text, we can go back the appropriate amount of physical
lines, and then start looking from there.  But if what's on display in
those N lines has little if any resemblance to what's in the buffer,
the only safe algorithm is to go to the beginning of the buffer and
start from there, which is prohibitively expensive.  So Emacs
implements several optimizations to make the operations reasonably
fast.  These optimizations only succeed when the display strings and
the text they cover are approximately similar in length, or if the
display strings are very short.  The specific optimization that caused
this bug was an attempt to be faster when text lines in a buffer are
very long.

> The overlays are also nested, so it might it be related to that?

In what way are they nested?  The problems happen only with overlays
that have the 'display' property.

> That said, your change seems to have fixed it.  

Thanks for testing.  I must ask, though: why do you need to use
display strings (rather than, say, invisible property) to do folding?