Re: [GNUnet-developers] Designing a gnunet directory app

[Christian Grothoff]

On Thursday 13 June 2002 10:19 am, you wrote:
> > You could also just create 'directories'. Insert a couple of files and
> > post the results of 'gnunet-search' (hash, length, crc, description) in a
> > 'directory' file. I would suggest to use 'directory' as the keyword for
> > these index-files for now. This way, we can use GNUnet itself instead of
> > a webpage as the indexing service.
> > And I'm sure, there'll be some code to create directories automatically
> > at some point. Any volunteers? :-)
>
> That would be within my capabilities and I'd be interested to
> code one. However, I suggest it to be done untypically
> this time: mainly, design first. Here's some things that came
> to mind.

Design first sounds good :-)

> There are several problems. First, a more general issue, how
> can we get rid of obsolete directories? Suppose I insert one
> with a keyword "directory" and include a timestamp. Could people
> querying for "directory" get that same hit even a year afterwards,
> as hitting it will keep it at least in some sense popular?
> Does gnunet have mechanism to get rid of files that are
> often 'hit' but rarely if ever downloaded?

No, it can't because search-results and downloads look identical in our 
encoding. For anonymity reasons, you don't want an adversary to be able to 
correlate hits. But it should not be a problem -- if the directory 
description contains some timestamp or other versioning information, a clever 
search-tool could see which are more/less recent directories. Note that any 
real possibility to 'update and destroy the old copy' is an open door for 
censorship (update with 'nothing' and destroy) and thus not acceptable. What 
could also be used is a signed directory that contains the key(word) for the 
update. Of course, everybody could publish another directory under the 
update-keyword, but not everybody could sign it with the same public key. 

> The next issue is, how should the app automate the creation
> of such list? Should just a short script be hacked together
> to direct the output of gnunet-search (on one or more runs)
> into a textfile and insert that? Or should a variant of
> gnunet-search be created thar would collect found keys to a
> file, and insert a new directory after a certain amount of
> new keys have been found?

I would rather add an option to 'gnunet-insert-multi' (gnunet-insert *has* 
all the information that gnunet-search prints) to create a directory when the 
files are inserted. 

> Is it benefical to store the actual
> keys, which can quickly disappear from the network?

No, the file-signature (hash, crc, size) are unique identifiers of the 
resource. Keywords (foo AND bar) can map to multiple different files.

> And what
> are the gains/losses against a solution where only some good
> keywords would be listed, perhaps with statistics of the time of
> list creation (e.g. keyword: test hits: 138 desc: miscellaneous
> junk for testing purposes)? That would just give people hints
> of what to look for, but different people rerunning those queries
> of popular words might be costly. In the end, of course, when
> gnunet encompasses the whole planet (and the colonies), a
> keywordlist will be unnecessary as *any key* will find plenty. ;D

Right, that's why I would for 'resource directories' that give descriptions 
and the unique resource identifiers. That is more helpful than keywords. In 
fact, we may find directories like that under keywords at some point (say 
you're searching for some Album of some Artist. What you get is a directory 
with the songs.

> Should compression be used on the directories?

I would make it an option to the user. Some directories will be too small to 
yield any big gains - compression was already thought as an option for 
gnunet-insert, but so far nobody did it and I still believe that the users 
can do it manually up-front if they really want to. The problem with 
compression for gnunet-insert is also that it conflicts with the on-demand 
encoding (indexing vs. insertion!), which would not apply for directories. 
Anyway, 'tar' can't be wrong, and there, it's an option :-)

> Naturally the directory listings should also be machine (eg gui)
> readable. The app should be able to retrieve newest up-to-date lists.
> Perhaps lists could be signed by sender, using a handle perhaps.

Right, we should have some standard format that allows signing with a 
pseudonym.

> Of course trivially done that would not be proof against attacks,
> but if most people are benevolent, it'd enable 'fan communities'
> to follow directories created by some famous persons. ;)

I don't see why it should not be done in a safe way - gnunet uses pretty much 
the strongest practical ciphers for signing certain messages, why shouldn't 
we do the same here? The time it takes to sign should be insignificant anyway.

> Anyway, the solution starts to remind me of frost on freenet,
> performing some of the issues mentioned above, essentially being a
> directory creation, maintaining and retrieval tool, all in
> one, packed with messaging capabilities.

I'm not sure what you mean with messaging capabilities, but looking into 
existing designs is definitely the right approach. I don't really know frost, 
any references? 

> Frost would be very easy
> to port to gnunet but it seems atleast to me that a straight
> or trivial port shouldn't be attempted before the issue
> is well thought about. Such "polling application" hasn't done
> much good to freenet, and I can't downright say what it
> would do to gnunet -- I don't know the guts of either system
> too well.

Polling? As in repeatedly query or what?

> The thing is though that if some similar app
> isn't done intelligently for gnunet, it will be eventually made
> brainlessly by a third party and bad things will happen (on freenet
> there's now atleast four ways to transmit and find files: private,
> not announced anywhere, "freesites", "frost" and "fmb". this
> has unnecessarily split the available resources between
> incompatible methods of announcement/retrieval/etc) :(

Well, I'm definitely for trying to estabish standards :-)

> Everyone, what do you think about these things? Any additional
> ideas? What should the dir app go for? Something like frost w/
> channels/interest groups, or just a small script to pack
> out found matches as ad hoc directory files?

I would see it as an extention to the gnunet-filesharing library that can 
then be used via options in the textui-clients or GUIs.

Christian