Problems with CurTerm and BROKEN_LINKER on Cygwin (and potentially other systems) with ncurses > 5.7

[Jean-Pierre Flori]

Dear all,

First, thanks for the great software.

We are including ncurses in Sage to replace termcap since the last release and it works great!

I've been recently trying to build Sage 5.12 on Cygwin and it seems ncurses is problematic with some nasty segfault occuring due to manipulation of null pointers when python is launched and loads readline which loads ncurses.

The full details are at http://trac.sagemath.org/ticket/15317 with further references (in particular it seems to happen not only on cygwin).

I've chased down the problem and it seems the problem is when

* BROKEN_LINKER = 1 (which is always the case on Cygwin...)

* and USE_REENTRANT = 0 (I've not tested USE_REENTRANT = 1 though but it seems it could work then).

Indeed, with that combination, CurTerm is set to _nc_prescreen._cur_term in ncurses/tinfo/lib_termcap.c and is never set to something else than 0 (whence the null pointer later on) in ncurses/tinfo/lib_cur_term.c.
Look especially around lines 89-93 where only with that combination of BROKEN_LINKER=1 and USE_REENTRANT=0 cur_term is set, but CurTerm is not (ok, if both are zero then CurTerm == cur_term).

Also setting CurTerm when BROKEN_LINKER=1 and USE_REENTRANT=0 makes the segfault go away, though I've done no extensive testing.

So maybe three cases should also be distinguished around these lines:

#if BROKEN_LINKER && !USE_REENTRANT
...

#elif BROKEN_LINKER || USE_REENTRANT

...

#else
...

#endif

as at the beginning of the file.

Anyway, I'm no ncurses expert, so my analysis may be wrong but it helped me so I prefer to keep some trace of it somewhere!

Final note: I had no problem building ncurses with GCC 4.8.2 and BROKEN_LINKER=0 on Cygwin.

So it may deserve new testing and not always set BROKEN_LINKER!

(Note though that the above bug was not Cygwin specific so something has to be done with CurTerm anyway.)

At least Python then launches; I've not made extensive testing though.

Best,

--
Jean-Pierre Flori