Re: [GNUnet-developers] Durable content-addressed/DHT storage

[Christian Grothoff]

On 01/02/2014 04:16 AM, David Timothy Strauss wrote:
> We maintain a FOSS file system [1] for users of our platform. One of
> the challenges, as we move move functionality into the client, is
> dealing with our copy-on-write volume-cloning. To get implicit
> copy-on-write, we reference SHA512 hashes of file content. Same
> content, same copy.
> 
> For a next-generation design, the DHT support in GNUnet would fit our
> use-case well.

I'm wondering why you're looking at the DHT, and not at FS for this.
Have you seen gnunet-fuse? (We used to have write-support in it, and
it would be interesting to see someone revive it.)

> But, I'm concerned about guaranteeing that a file is
> available within the sort of F2F networks we'd be setting up. Is there
> any way to guarantee that, aside from republishing constantly?

If your nodes are trustworthy and are not experiencing churn, only the
expiration time limits how long data stays around.  So your use-case
(and specifically, environment) largely determines how often you have
to republish to ensure availability.

> Is
> there interest in support for secondary backends, like S3 (with or
> without encryption of the content), in order to provide arbitrarily
> large storage?

The DHT uses a subsystem "DATACACHE" for storage, and FS uses
"DATASTORE".  Both have a pluggable backend (sqlite, mysql, postgres)
and it should be simple to add support for other databases.  You can
find the API(s) that you'd have to implement here:

https://gnunet.org/svn/gnunet/src/include/gnunet_datastore_plugin.h
https://gnunet.org/svn/gnunet/src/include/gnunet_datacache_plugin.h


So the easiest way to do this might be to (1) add an S3 plugin to
datastore/datacache, and then (2) extend gnunet-fuse with CoW
support.  A hard design question for us was always how to nicely
return the URI (or file-system mount point) to the user that resulted
from the copy.  If you have a good answer for how this should be
done (essentially, how do you _nicely_ tell the user the top-level
identifier after umount), that'd be interesting.

Happy hacking!

Christian