GNU tar --append is too slow

[Adye, TJ (Tim)]

Hi,

In switching from Solaris' tar to GNU tar, I've noticed one significant
drawback. Appending files to an existing archive becomes very slow with GNU
tar when the tar file grows large. With Solaris tar there is no extra
overhead. Here are a few timings (mins:secs real time, Solaris then GNU tar)

Create a 1.5GB tar file: 1:48,   1:46
Append a 1 byte file:    0:00.1, 0:47

I am using the same commands in each case

  tar cf archive.tar 1536mb.dat
  tar rf archive.tar 1byte.dat

both on a Solaris 8 system with the native tar and GNU tar 1.4 (all files on
local disk). I saw a similar effect with GNU tar 1.3 and 1.4 on Solaris and
RedHat Linux 7.2. I don't think it is an effect of the tar file format,
since I see the same append times regardless of which tar program initially
created the tar file. I did get varying times for different tests
(presumably depending on what was already cached), but always saw
significant times with GNU tar adding the tiny file to the end of a large
tar file, whereas Solaris tar performed this operation in negligable time.

I guess that Solaris tar skips straight to the end of the file, while GNU
tar reads through the file before appending. Is there any way to obtain the
faster response with GNU tar? I've experimented with a number of tar
options, but perhaps there's some magic option. Or is this something that
could be fixed in a future version?

We are using tar to group small files before writing them to our mass-store
systems and would like to move to using Linux and GNU tar. Unfortunately the
performance hit described above would cripplingly reduce our overall
transfer times from ~10 GB/hour to ~1 GB/hour. We have typically 2-200 MB
files, 1.9 GB tar files, and ~10 TB total to write per week (multiple
parallel jobs/servers). It would take a significant re-engineering to write
the files to the archive all at once, rather than using tar --append.

Can you advise?

Thanks,
Tim.

==============================  cut here  ==============================
Tim Adye, BaBar Group, Particle Physics Dept.,             _   /|
          Rutherford Appleton Laboratory, UK.              \'o.O'   Oop!
e-mail:   address@hidden                                =(___)=  Ack!
WWW:      http://hepwww.rl.ac.uk/Delphi/Adye/homepage.html    U  Thphft!