Re: [lmi] Building shared zlib

[Greg Chicares]

On 2017-08-29 17:54, Vadim Zeitlin wrote:
> On Tue, 29 Aug 2017 15:09:36 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2016-07-15 21:11, Vadim Zeitlin wrote:
> GC> > On Fri, 15 Jul 2016 21:04:16 +0000 Greg Chicares <address@hidden> wrote:
> GC> 
> GC> [...problems building zlib for use with libxml2, which now reappear--see 
> below...]
> GC> 
> GC> > GC> Where do we go from here?
> GC> > GC>  (1) Use './configure && make' and fix up the problems manually?
> GC> > GC>  (2) Try to fix the autotoolization problem?
[...]
> GC> > GC>  (3) Use the msw-specific makefile provided by zlib?
> GC> > GC>  (4) Just use the zlib.org msw binaries?
[...]
> GC> Before presenting the latest problems, let me ask whether there are
> GC> other, better options:
> GC> 
> GC> (5) Use a zlib built by wx
> GC> (6) Use zlib binaries provided by the MinGW-w64 maintainers
> GC> (7) Drop zlib and use lzma instead
> 
>  I think that if we stay with zlib, it would really make sense to somehow
> reuse the version already bundled with wxWidgets. It seems wasteful to use
> 2 instances of the same library in the program and, in fact, it could be
> actively harmful if we end up with different versions of the same symbols
> in the same binary due to this.

Agreed, iff we stay with zlib, because if something is broken in wx the
maintainers fix it. (Would that all free-software projects were such.)
But it's a black hole.

My aversion to zlib is that, for all the time I've spent on it, I can't
cross-build a zlib DLL in my chroot, and I can't build one in cygwin.
I seem to have such a DLL in one cygwin VM, but I don't know where it
came from and I don't know how to recreate it. I'm pretty well convinced
that it's time to cut our losses and try something else instead.

> GC> The principal motivation, as explained here:
> GC>   http://lists.nongnu.org/archive/html/lmi/2016-06/msg00046.html
> GC> is to use compression as (weak) encryption, and lzma would serve
> GC> that purpose just as well as zlib. Speed shouldn't be a problem:
> GC>   https://tukaani.org/lzma/benchmarks.html
> GC> | Decompression  In terms of speed, gzip is the winner again.
> GC> | lzma comes right behind it two to three times slower than gzip.
> 
>  To be honest, I don't see any real need to bring in LZMA. It does have a
> much better compression, of course, so if we cared about this, it would be
> almost obviously the best choice. However we don't, whereas speed is
> important and zlib is, of course, much faster: "right behind" is weird when
> it's followed by "two to three times slower" and more recent benchmarks
> like the one here which I just found
> 
> https://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO
> 
> show that the factor is more like 4-8 for compression (but for
> decompression it's indeed barely 2). Anyhow, if we really cared about
> performance, we'd use LZO, LZF or Snappy, but even if we probably don't,
> much, it's still at best an absence of argument in LZMA favour.

Let's measure it with 'sample.database', which is one of the largest
files we'd be compressing (the largest one being only 15% larger):

/tmp[0]$time gzip --stdout /opt/lmi/data/sample.database >gz.database 
gzip --stdout /opt/lmi/data/sample.database > gz.database  0.00s user 0.00s 
system 0% cpu 0.004 total
/tmp[0]$time xz --stdout --threads=32 /opt/lmi/data/sample.database 
>xz.database 
xz --stdout --threads=32 /opt/lmi/data/sample.database > xz.database  0.03s 
user 0.00s system 89% cpu 0.036 total

Compression took nine times as long, but...

/tmp[0]$time gzip -cdk gz.database >/dev/null 
gzip -cdk gz.database > /dev/null  0.00s user 0.00s system 0% cpu 0.003 total
/tmp[0]$time xz -cdk xz.database >/dev/null 
xz -cdk xz.database > /dev/null  0.00s user 0.00s system 0% cpu 0.003 total

...decompression speeds are actually the same. The decision matrix is:

  xz gzip  criterion
 --- ----  ---------
 100  100  speed as % of gzip
 yes   no  do we have a reliable way to build it?
   2   20  estimated hours already spent trying to make it work
   0  INF  remaining hours of effort required to make it work

BTW, the file sizes are:

/tmp[0]$ls -l gz.database xz.database  /opt/lmi/data/sample.database
-rw-r--r-- 1 greg greg 103899 Aug 29 23:56 /opt/lmi/data/sample.database
-rw-r--r-- 1 greg greg   5060 Aug 30 00:43 gz.database
-rw-r--r-- 1 greg greg   4188 Aug 30 00:43 xz.database

> GC> Option (7) sounds most promising in light of all the difficulties
> GC> we've had with zlib
> 
>  The fact that we had some build problems with zlib doesn't mean we won't
> have any with lzma/xz. This is a much newer library, of course, so I'd
> expect to have fewer problems, but OTOH zlib is (still) much more
> widespread so I'd also expect the solutions for any problems with it to be
> more readily available. There is also the fact that zlib is a very simple
> library with not much code in it at all, while lzma is much less so.

Testing with the compressed files generated above, I already have
xz + libxml2 working:

/tmp[0]$wine xmllint --encode ASCII xz.database |wc
   5067    5091  106750

but I haven't succeeded with gz + libxml2:

/tmp[0]$wine xmllint --encode ASCII gz.database |wc
gz.database:1: parser error : Start tag expected, '<' not found
/��Y
^
      0       0       0

There's one remaining problem, though: it doesn't work with lmi.
I rebuilt lmi after building libxml2 with xz (and the example
above proves that it works), but I get the same error message:
  Unable to parse xml stream: Start tag expected, '<' not found
  [file /opt/lmi/src/lmi/xml_lmi.cpp, line 127]
with a compressed input file, no matter which algorithm I use
to compress it. I tried this with copies of 'sample.ill' from
the lmi repository, compressed with the same sort of commands
used above:

-rw-r--r-- 1 greg greg 2460 Aug 29 23:59 gz.ill
-rw-r--r-- 1 greg greg 2432 Aug 29 23:59 xz.ill

Any idea what's wrong? Do I need a later version of xmlwrapp?
Would xmlwrapp need to be modified to use liblzma? I wouldn't
think so, but this "project" file:
  https://github.com/vslavik/xmlwrapp/blob/master/platform/Win32/master.proj
mentions zlib specifically and not lzma.