Re: Uncompressed mbox format storage

[Bob Proulx]

wrotycz wrote:
>   I am surprised why is the mbox format archives uncompressed. Considering
>   that almost everything is compressed, and not only 'these days', I wonder
>   what is the rationale for that.

Mailman requires it to append new messages into the mailbox.  That's
the main reason.

Many file systems these days have built in compression.  I realize
that the file system on lists.gnu.org does not.  But it could.  And
then compressing it on top of a compressed file system is redundant.

For the most part the file are not compressed because all of the tools
which work with those files on the server side need the files to be
uncompressed.  I am sure that a lot of random and unknown to us tools
would need to be modified if it were switched to a compressed format.
That's a lot of effort.  And again IMNHO the effort would be better
spent moving those files to a compressed file system if one wanted to
do that.

As far as network download bandwidth goes, not that many people
download those files.  That hasn't been the main bandwidth drain when
I have looked at things.  Also web servers can offer content
compressed for the transmission.  That's where I would put the effort
in if network bandwidth were to be reduced.

I will leave further discussion of this to Ian who maintains the
mailing list server at this level.  I am sure Ian will have more
insight into this than I do.

Bob

>   Wouldnt it be better compressed sand save ~70% of storage and same amount
>   of bandwidth?
>   I did a test (results below) and the difference is quite remarkable.
>   ~~~
>     size    name                          ratio
>     7831479 bug-standards.cat
>     2299399 bug-standards-bz2.cat
>     2287163 bug-standards-gz.cat          29.20%
>     2054411 bug-standards-lzma.cat
>   96964420 coreutils.cat
>   28494935 coreutils-gz.cat              29.38%
>   25667675 coreutils-bz2.cat
>   22314227 coreutils-lzma.cat
>   60940336 help-bash.cat
>   20653756 help-bash-gz.cat              33.89%
>   19524730 help-bash-bz2.cat
>   16975032 help-bash-lzma.cat
>      456171 listhelper-discuss.cat
>      118833 listhelper-discuss-bz2.cat
>      108261 listhelper-discuss-gz.cat     23.73%
>      102425 listhelper-discuss-lzma.cat
>     8290405 mailman.cat
>     1314314 mailman-gz.cat                15.85%
>     1076689 mailman-bz2.cat
>      886116 mailman-lzma.cat
>   ~~~