Re: [GNUnet-developers] Idea for file storage in GNUnet

[hypothesys]

Dear LRN and Christian,

Thank you for your replies :). Regarding latency, peers running close to
quota and cached blocks I may, of course, not be understanding something but
still believe this could work.

First of all an ignorance-derived question regarding LRN point that the
probability of finding a block of data in a random node not sharing it on
purpose would be small. Would this probability not also be additive (or
increase exponentially?) when the said random node, after checking against
the local data block index and not find anything, relayed on the data
request? Assuming the data distribution in the network is not too
non-uniform/asymmetrical I do not see why that would not be the case.

Also, privacy issues derived from prioritizing downloading and publishing
blocks to the "normal/minimum threshold/allocated" data storage: Why would
this be necessary? GNUnet storage could operate as normal and this new
"dynamic/maximum threshold" storage serve not only to cache "hot" and
popular blocks, but also use a percentage of the dynamic storage to cache
copies of other nodes "normal/minimum threshold/allocated" storage. In this
way both latency AND censorship-resistance would improve. It would probably
need a mechanism to re-organize/swap files in normal storage depending on
the priority/distribution of files throughout the network though.

I may be making a gross oversimplification here but it feels as if this
increased dynamic storage would add "capacity/ability/versatility" to
GNUnet, which could in turn, depending on the implementation, be used to
boost one, or several at the same time, feature(s) of GNUnet.

Once again if I am wrong please say so. Not from the field ;)

Cheers,

hypothesys



Christian Grothoff-6 wrote:
> 
> On 12/07/2012 08:34 AM, LRN wrote:
>> And "spare" is the problem. I can easily spare 20 or 40 gigabytes, but
>> 100 or 200 is somewhat trickier. I might have that kind of space now,
>> and be willing to give it to GNUnet, but i might want that space back
>> at some point. Not sure what GNUnet will do right now, if i shut down
>> my node, reduce the datastore size, then start the node up again.
>> Probably discard lowest-priority blocks until datastore shrinks to the
>> new limit?
> 
> Yes, that's what it would do.  However, there is a caveat: the
> mysql/sqlite/postgres database that is involved might be happy to delete
> the records, but might not automatically reduce its file system space
> consumption.  So you may have to additionally trigger some
> database-specific routine to force the database to defragment/relinquish
> its allocation/garbage collect/whatever.
> 
> Doing this may (temporarily) double your space requirements, depending
> on the database.  So this is an "implementation detail" that would make
> an automatic 'shrink if disk is full' implementation somewhat harder
> (but likely not impossible, as you can predict the necessary space for
> the reorganization).  Alternatively, one _may_ be able to use multiple
> database files and just delete one of those entirely once the quota is
> reached (this depends on the database backend that is being used).
> 
>> Now, having a minimum space allocated to the datastore, and then just
>> using N% of the remaining free disk space for for datastore too, while
>> it's available - that really makes the decision easier. If GNUnet is
>> then taught to use pre-allocated datastore for important blocks (files
>> being downloaded or published; what are privacy issues here?), that
>> would mean that your node will serve _your_ interests first, and will
>> use the free space available to serve the network as best as it can.
> 
> I don't think there is a problem here.  We already have routines to
> shrink-to-quota which are triggered if we are above quota (due to
> additional insertions or due to quota being lowered).
> 
>> It should maintain either F% of space free, or G gigabytes (whichever
>> is larger). Obviously, F and G are configurable (i.d say - default F
>> to 20, and G to 20; unless GNUnet daemon that would reclaim free space
>> would be a slowpoke, 20 gigabytes should give it enough time to react).
>> It should also be completely disabled for SSDs, IMO. Because they are
>> small to begin with, _and_ because their performance degrades greatly
>> as they are filled with data.
> 
> I suspect those arguments may not hold for long as SSD technology
> progresses...
> 
>> Thus the idea is the same as with CPU resources - you set up low and
>> high thresholds for CPU load that GNUnet can cause. It will go as high
>> as the high threshold when uncontested, and will go down to the low
>> threshold when other processes compete for CPU resources with GNUnet.
>> Same for storage - use large portion of available free space for
>> datastore (primarily - for migrated and cached blocks), but be ready
>> to discard all that, and go as low as the size of the pre-allocated
>> datastore.
> 
> Well, LRN, if you think peers actually run close to quota, there is a
> nice GNUNET_UTIL-call for starters: GNUNET_DISK_get_blocks_available.
> Adjusting the quota option in datastore based on that should not be too
> hard for you; the real bitch will be testing the various backends to
> make sure that they actually reduce disk space consumption --- and I
> guess reliably finding out which partition MySQL/Postgres actually store
> their data on might also be not so easy...
> 
> 
> Happy hacking! ;-).
> 
> -Christian
> 
> _______________________________________________
> GNUnet-developers mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnunet-developers
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Idea-for-file-storage-in-GNUnet-tp34768221p34770670.html
Sent from the GnuNet - Dev mailing list archive at Nabble.com.