[gnuastro-devel] [task #14191] Enable FITS input and output using Lzip

[Antonio Diaz Diaz]

Follow-up Comment #3, task #14191 (project gnuastro):

Mohammad has found an article
<http://iopscience.iop.org/article/10.1086/599023> that explains why using
gzip (and by extension bzip2 and lzip) to compress images is in general a bad
idea. Gzip, bzip2 and lzip see the data as strings of bytes, not as pixels:

"The most important distinguishing characteristic of GZIP compared to the
other compression algorithms used in this study is that GZIP treats each 8-bit
byte of the input data stream as an independent datum, whereas the other
compression methods operate on the numerical value of the input image pixels
as multibyte quantities. This puts GZIP at a distinct disadvantage when
compressing astronomical images with 16-bit or 32-bit pixel values because,
unlike the Rice and Hcompress algorithms, GZIP cannot use the numerical
difference between adjacent pixels as a means of improving the compression."

As a consequence, gzip, bzip2 and lzip may be not very useful to compress data
masks because:

"In many cases, choosing the algorithm that produces the very highest
compression ratio is of secondary importance because the masks compress so
well with any algorithm that the size is insignificant compared to the rest of
the associated data set. (...) In practice, it may be simplest to just use the
same compression algorithm on the data mask as is used on the associated
astronomical image."

Unless we can prove that lzip can compress binary tables or floating point
images significantly better than the existing methods, I think we should not
propose the addition of lzip to the fits standard because:

"Since the compression ratios produced by Rice and Hcompress are already
within 75% to 90% of an ideal algorithm with K = 0, any further improvements
to the compression algorithms will produce relatively little gain in the size
of the compressed images."

The findings of the article suggest that maybe gzip tiled compression should
not have been added to the fits standard either:

"Overall, the Rice compression algorithm strikes the best balance of
compression and computational efficiency; it is 2-3 times faster and produces
about 1.4 times greater compression than GZIP".


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/task/?14191>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/