Re: [GNUnet-developers] high disk-io

[Benjamin Kay]

On Monday 05 April 2004 05:32 pm, Christian Grothoff wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> > Conditions under which gnunetd is running here:
> > MySQL (1024MB)
> > free sais 100MB cached (its a small machine)
> > 90000 (set as up & down bandwidth in config file)
> >
> >  I think there should be similar way of control be used for disk-io like
> > for CPU and network load. To reduce padding at first and if it is not
> > enough to reduce even query lookups.
>
> Actually, there is one additional possibility, which is to make the disk-io
> less random.  If, for example, we gather more than just one block at a time
> from a file and instead read a dozen or two (possibly in sequence), that
> does not increase IO much but improves throughput dramatically.  The bad
> news is that it also makes deniability worse since chances of a peer that
> does not have the file pushing out closely related blocks from a presumably
> 'random' assembly of migrated blocks are low.  Anyway, together with the
> buffering of content by the active-migration thread (may cost memory) it
> should be possible to achieve an acceptable trade-off that can still be
> better than just not doing migration.

I don't suppose the migrated content could be "defragmented" like a 
journalizing filesystem so that closely related blocks are contiguous (I 
think mysql can do something like this). If all, or almost all, nodes did 
this journalizing, then wouldn't the behavior described above appear fairly 
"normal" - restoring deniability? Or put another way, do the migrated blocks 
need to be random?

> But for all three approaches to lower IO, we first need some code to
> measure how much (actual) IO is going on -- otherwise the code would not
> know when to throttle.

Yeah. Sounds like a long involved algorithm in the making. (smacks self)

> Thanks for the data-point, I'll definitely keep it in mind (though I have
> my doubts that it holds for machines with less bandwidth, but then again,
> those may also not have problems with the IO load...)

Even with my relatively low bandwith, HD consumption is still an issue - 
especially since the GNUnet db is on the same HD as the rest of my 
applications. I think that users could actually benefit from this, albiet 
they could benefit more from other improvements.

-Benjamin Kay