Re: What are best lzip compression options for large game tar archives?

[Antonio Diaz Diaz]

Hi Hoël,

Hoël Bézier wrote:
> Since most distribution packages are tar archives compressed with
> different algorithms (Debian favors bzip2 or xz, Arch favors zstd for
> example), I thought that where possible I could as well provide support
> for lzip compression (mainly for Gentoo actually, as one can write its
> own support for other compression tools) because I do like a lot the way
> lzip was designed, and game packages might be kept around for quite a
> few years (I still have a tar archive of a Warcraft III installation
> hanging around for at least seven years).

Thanks. I think that lzip is indeed a good choice for game packages. :-)


> However, since we compress fairly big packages (most popular games
> nowadays are at least 1GB big), current compression with lzip
> (implemented through tar's --lzip switch) is quite slow. For example,
> packaging bastion, which is 1.1G big, took half an hour on my computer.
>
> Since I have seen that there were several options to lzip that could
> reduce compression time or improve compression ratio, I was wondering if
> you could tell me which options would be most fit for my purpose, and
> with which caveats, if any.

You may use option -0, which compresses about as fast as gzip:

http://www.nongnu.org/lzip/lzip_benchmark.html#gzip

But for compression of large files on multiprocessor machines, plzip can be 
much faster than lzip at the cost of a slightly reduced compression ratio. 
Using option '-0' with 64 compression threads, plzip may be even faster (515 
MB/s) than pigz:

http://www.nongnu.org/lzip/plzip_benchmark.html

So you may try a command like

  tar -c foo | plzip -0vvo foo.tar.lz

or with a recent version of GNU tar you can use

  tar -cf foo.tar.lz -I 'plzip -0' foo

Best regards,
Antonio.