Re: [Help-gnunet] Serious problem with insert

[Christian Grothoff]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nah, this is well-known. The reason is, that for random data, GNUnet get's 
random hashes and must do many random disc-accesses to insert the file. For 
/dev/zero, the hashes of all blocks are identical and thus the disk-accesses 
all go to the same location (in fact, only (j+2) kb are stored for a file of 
size 51^j). 

The degradation over time for the random case comes from caching effects. For 
very small amounts of data, the Kernel/the drive can cache these and defer 
writing until after gnunet-insert returns. Thus this is more an artefact of 
your measurement technique. 

Insertion of files into GNUnet is, by design, of linear complexity.

Christian

On Wednesday 04 September 2002 03:14 am, Andrew Hart wrote:
> I think the following shows that gnunet-insert has serious problems:
>
> address@hidden poo]$ time gnunet-insert -f 0Mfile "rubbish" rubbish
> File 0Mfile successfully indexed.
>
> real  0m0.211s
> user  0m0.050s
> sys   0m0.030s
> address@hidden poo]$ time gnunet-insert -f 10Mfile "rubbish" rubbish
> File 10Mfile successfully indexed.
>
> real  0m11.295s
> user  0m2.270s
> sys   0m0.970s
> address@hidden poo]$ time gnunet-insert -f 20Mfile "rubbish" rubbish
> File 20Mfile successfully indexed.
>
> real  0m46.157s
> user  0m4.360s
> sys   0m1.450s
> address@hidden poo]$ time gnunet-insert -f 30Mfile "rubbish" rubbish
> File 30Mfile successfully indexed.
>
> real  4m51.883s
> user  0m6.290s
> sys   0m2.580s
> address@hidden poo]$
>
> Note: the ??Mfile is the first ??M of Star Wars Episode II.divx.avi and is
> therefore pretty random data.
>
> Now compare to the results for ??Mzeros (from /dev/zero):
> address@hidden poo]$ time gnunet-insert -f 0Mzeros "zeros" zeros
> File 0Mzeros successfully indexed.
>
> real  0m0.588s
> user  0m0.050s
> sys   0m0.040s
> address@hidden poo]$ time gnunet-insert -f 10Mzeros "zeros" zeros
> File 10Mzeros successfully indexed.
>
> real  0m5.172s
> user  0m2.210s
> sys   0m0.480s
> address@hidden poo]$ time gnunet-insert -f 20Mzeros "zeros" zeros
> File 20Mzeros successfully indexed.
>
> real  0m10.029s
> user  0m3.970s
> sys   0m1.090s
> address@hidden poo]$ time gnunet-insert -f 30Mzeros "zeros" zeros
> File 30Mzeros successfully indexed.
>
> real  0m15.239s
> user  0m6.190s
> sys   0m1.490s
> address@hidden poo]$
>
>
> So gnunet-insert is rather good at inserting all zeros and time is
> proportional to file size.  However, when real compressed data is used,
> time grows exponentially(?).  This really means that some sort of file
> spliting is needed and unfortunately the practical block size is only 10M
>
>
> _______________________________________________
> Help-gnunet mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/help-gnunet
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9dhic9tNtMeXQLkIRAoAlAJwNaXd0ROGu+v2dm9PHMEm34+2YBACfRTv8
UrsBcyz4q3aWLqUIR5EmG5o=
=7YYE
-----END PGP SIGNATURE-----