Re: [lmi] Testing build with native MinGW 4.9.1

[Greg Chicares]

On 2016-01-20 21:16, Vadim Zeitlin wrote:
> On Wed, 20 Jan 2016 18:00:44 +0000 Greg Chicares <address@hidden> wrote:
>
> [... libxml2 'testReader' segfaults (so programs like lmi that use
>  libxml2 segfault immediately), and the problem seems related to zlib...]
>
> GC> Wow. I'm very grateful that you found that problem and solved it.
> 
>  I was so worried by not understanding what it was due to, that I spent
> even more time and I think I finally did find the reason: in some

[Time well spent, because now we can learn from this experience. Thanks.]

> prehistoric times (definitely before zlib 1.2.3 released 10.5 years ago and
> maybe for quite some time longer), z_stream type used by zlib IO functions
> and gzFile type returned by gzopen() was one and the same or, at least,
> interchangeable and libxml2 abused this and used the latter as the former.
> This doesn't work at all in any more or less recent zlib versions (and this
> is what I saw under debugger and which made me doubt the integrity of my
> build) because the types are completely different and it was fixed in
> libxml2 itself 6 years ago, see
> 
> https://github.com/GNOME/libxml2/commit/a7e79f28689c574e0bbef17f4cb3da00249181ff
> 
> So this explains the problem: MinGW-w64 4.9.1 comes with zlib 1.2.8 and
> without the fix above, libxml2 can't work with it.

Hmmm. Our old mingw.org toolchain had no zlib:
  /opt/lmi/MinGW-20090203-mingw-org[0]$ls **/zlib*
  zsh: no matches found: **/zlib*
  /opt/lmi/MinGW-20090203-mingw-org[1]$ls **/libz*
  zsh: no matches found: **/libz*
but the new MinGW-w64 toolchain provides it (who ordered that?):
  /MinGW_[0]$ls **/libz*
  i686-w64-mingw32/lib/libz.a
(I wonder why we never needed '--without-zlib' before, even though
we were building without zlib. I guess this option means "exclude
zlib even if it's detected (and if it's not detected, it'll be
excluded silently even without this option)".

There's nothing wrong with using old libraries for more than a
decade--RHEL LTS does it:

https://access.redhat.com/support/policy/updates/errata
| Red Hat Enterprise Linux versions 5, 6, and 7 each deliver 10 years
| of support in Production Phases 1, 2, and 3 followed by an Extended
| Life Phase.

but AIUI they wouldn't update to a new compiler version that adds
who-knows-what unasked-for libraries. Here, lmi's LTS approach got
tripped up by MinGW-w64's rolling release; they don't provide libxml2,
but for some reason they do provide zlib.

At least it's zlib-1.2.8:
  /MinGW_[0]$head -2 i686-w64-mingw32/include/zlib.h
  /* zlib.h -- interface of the 'zlib' general purpose compression library
    version 1.2.8, April 28th, 2013
which is the latest release according to
  http://www.zlib.net/

Now that we have zlib, can any other library (wx, for instance) make
good use of it? Or, in a world where bzip2 is obsolescent and md5sums
are obsolete, does it make any sense to use zlib? It's just LZ77 AFAICT,
contemporaneous with "Muskrat Love", and do compression algorithms age
better than pop music?

> FWIW I also checked that
> the latest libxml2 can be built without neither --without-threads nor
> --without-zlib and still works correctly.

Well...since you've already done the work to verify that...and since
we're already rebuilding libxml2 and libxslt and therefore need to
test them both anyway...does it make sense to upgrade both those
libraries now? And which versions would we use, given that I'd like
to cross-compile from debian-7?

https://packages.debian.org/wheezy/libxml2
| libxml2 (2.8.0+dfsg1-7+wheezy5) [security]

http://www.xmlsoft.org/news.html
| v2.9.3: Nov 20 2015 [latest]
| 2.8.0: May 23 2012
|   Features: add lzma compression support (Anders F Bjorklund)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (Does that answer my zlib question above?)
| 2.6.26: Jun 6 2006
  ^^^^^^ that's what we're using today

I guess the debian security updates would take care of most of
the emerging issues. Maybe not all: I see CVE-2015-8242 here
  http://www.xmlsoft.org/news.html
but not here
  
http://metadata.ftp-master.debian.org/changelogs/main/libx/libxml2/libxml2_2.8.0+dfsg1-7+wheezy5_changelog
However, we're using msw-built binaries in production and I don't
imagine xmlsoft.org provides security backpatches. We *could*
force debian-7 to use a later version, but I want a stable LTS OS,
not a chimera.

OTOH, end users install lmi in its own isolated directory, and lmi
code isn't network-aware, so do all those CVEs matter a whole lot?
Would any malefactor ever even think of mounting a "billion laughs
attack" on lmi? If we're already updating from 2006 to 2012, is
there a compelling reason to go all the way to 2015 even though it
makes cross-compiling from debian-7 more difficult?

>  The only still remaining question is why did I need to apply the patch to
> ensure that the alert functions are linked in with the Debian
> cross-compiler but not the native compiler. I'm not sure if it's worth
> debugging this further, however, as it's clearly an
> implementation-dependent optimization anyhow. It would be better to just
> apply the patch to make sure the problem can never arise rather than
> hoping that it doesn't.

So...when cross compiling from debian, you received a build-time error?
Those alert functions certainly do work with the 'install_msw.sh' lmi
binary I just built; I verified that from its "Test" menu.

I've always insisted that the "volatile bool ensure_setup" technique is
guaranteed to work by the C++ standard; OTOH, 'force_linking.hpp' states
its purpose thus:

/// Immunize the present TU against an "optimizing" linker.

and the language standard imposes no requirements on linkers. That
header's from wx, so it's been broadly tested; its author is Vaclav, so
it's well engineered; we already use it for other purposes; and the
macros have such beautiful names...what's not to like about this change?

I wondered why we did this already for four other facilities but not
for 'alert', but our reasoning is explained here:

http://lists.nongnu.org/archive/html/lmi/2014-10/msg00007.html

Okay...so at first we didn't need it, then we needed it in some cases
but not others because of one particular linker, and now we need it in
one more case because of one more linker, so...should we just take care
of all cases presumptively now, and never revisit this again? I guess
our universe of consideration is the cross product of
  {file_command,progress_meter,system_command,pdf-thing | 
alert,mc_enum_types_aux}
    (components listed left of '|' already handled)
and
  {wx, cgi, cli}
    (only wx handled; the others might use libraries someday)
so should we now add 'mc_enum_types_aux' as well as 'alert', and write
LMI_FORCE_LINKING_EX_SITU in 'main_c[gl]i.cpp' too? It looks so simple
that I think I'll just do it.