Re: [PATCH] dist-xz: don't hard-code -9: honor setting of XZ_OPT

[Lasse Collin]

On 2010-10-03 Ralf Wildenhues wrote:
> * Jim Meyering wrote on Sun, Oct 03, 2010 at 02:56:21PM CEST:
> > The name I used there, XZ_OPT, is dictated by the tool.  An
> > alternative is to invoke xz with no explicit XZ_OPT setting, but
> > that would result in a semantic change: using the default
> > compression level of -7 rather than what was hard-coded as -9. 

The default is -6. In LZMA Utils it is -7, but due to differences in 
presets, -6 in XZ Utils is quite similar to -7 in LZMA Utils.

> > Perhaps that would be better, because some people have complained
> > that using -9 makes it harder to decompress (requires more memory)
> > on extremely low-memory (embedded) systems.

Low memory doesn't necessarily imply an embedded device. When 
decompressing a source package, it's more likely a desktop computer from 
1990s, I think. Yes, some people do use them in 2010.

It's a different question if it makes sense to take these systems into 
account when creating source packages. While .xz files can be 
decompressed fine on such systems as long as the compression settings 
haven't been too high, it may be better to just ask those people to use 
.bz2 or .gz version of the source package. The problem is that some 
people use a package manager that takes care of downloading the source 
files, and it doesn't necessarily support changing which file formats 
are preferred.

> > There is no point in changing lzma, since it's deprecated.
> 
> Except that the legacy lzma Utils that I tested on one MSYS install
> failed with OOM due to the -9, causing a spurious testsuite failure.

With a recent xz snapshot, it is true with xz too.

> Wrt. xz: I think the general approach for 'dist-*' should be: by
> default use highest compression that is portable.  Rationale: the
> default behavior is one compression plus upload, and lots of
> downloads and decompressions.  That it makes testing the package
> take longer is a QoI issue which the developer can tune with the
> XZ_OPT knob then.  OTOH, when we find out that -9 fails due to OOM
> on some systems (and I have no idea whether the lzma utils failure
> corresponds to a similar xz failure) then we should consider backing
> down to something lower.  I really prefer the defaults to work OOTB.

xz used to adjust the compression settings automatically down when the 
system had somewhat low amount of RAM, but it has now been disabled as 
part of a change affecting decompressor memory usage limiting. I still 
don't know if I should restore the limiting & adjusting by default for 
compression (and only for that). It currently can be enabled e.g. via 
the XZ_DEFAULTS environment variable, but I guess most users won't do 
that before something has already gone wrong.

The default level -6 needs 94 MiB of memory. That probably will work 
fine for almost everyone. -9 needs 674 MiB, and there will be many users 
for which it's too high. Also, -7, -8, and -9 are useful only when the 
uncompressed file is bigger than 8 MiB, 16 MiB, and 32 MiB, 
respectively. Using a high setting for tiny files is waste of memory. On 
the other hand, -e (--extreme) might help with small files too.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode