[Help-gnunet] Serious problem with insert

[Andrew Hart]

I think the following shows that gnunet-insert has serious problems:

address@hidden poo]$ time gnunet-insert -f 0Mfile "rubbish" rubbish
File 0Mfile successfully indexed.

real    0m0.211s
user    0m0.050s
sys     0m0.030s
address@hidden poo]$ time gnunet-insert -f 10Mfile "rubbish" rubbish
File 10Mfile successfully indexed.

real    0m11.295s
user    0m2.270s
sys     0m0.970s
address@hidden poo]$ time gnunet-insert -f 20Mfile "rubbish" rubbish
File 20Mfile successfully indexed.

real    0m46.157s
user    0m4.360s
sys     0m1.450s
address@hidden poo]$ time gnunet-insert -f 30Mfile "rubbish" rubbish
File 30Mfile successfully indexed.

real    4m51.883s
user    0m6.290s
sys     0m2.580s
address@hidden poo]$ 

Note: the ??Mfile is the first ??M of Star Wars Episode II.divx.avi and is 
therefore pretty random data.

Now compare to the results for ??Mzeros (from /dev/zero):
address@hidden poo]$ time gnunet-insert -f 0Mzeros "zeros" zeros
File 0Mzeros successfully indexed.

real    0m0.588s
user    0m0.050s
sys     0m0.040s
address@hidden poo]$ time gnunet-insert -f 10Mzeros "zeros" zeros
File 10Mzeros successfully indexed.

real    0m5.172s
user    0m2.210s
sys     0m0.480s
address@hidden poo]$ time gnunet-insert -f 20Mzeros "zeros" zeros
File 20Mzeros successfully indexed.

real    0m10.029s
user    0m3.970s
sys     0m1.090s
address@hidden poo]$ time gnunet-insert -f 30Mzeros "zeros" zeros
File 30Mzeros successfully indexed.

real    0m15.239s
user    0m6.190s
sys     0m1.490s
address@hidden poo]$ 


So gnunet-insert is rather good at inserting all zeros and time is 
proportional to file size.  However, when real compressed data is used, time 
grows exponentially(?).  This really means that some sort of file spliting is 
needed and unfortunately the practical block size is only 10M