[GNUnet-developers] Re: [p2p-hackers] Help needed: Autonomous NAT traversal test [Was: enabling bridges on NATed clients]

[Stephen Williams]

I've been lurking for a while.  Apologies that I've missed most of the 
conversation.

Michael Blizek wrote:
> Hi!
>
> On 16:35 Sun 07 Mar     , Eugen Leitl wrote:
>   
> > ----- Forwarded message from Christian Grothoff <address@hidden> -----
> >
> > From: Christian Grothoff <address@hidden>
> > Date: Sun, 7 Mar 2010 12:30:53 +0100
> > To: address@hidden, address@hidden, address@hidden,
> >         address@hidden, address@hidden
> > Subject: Help needed: Autonomous NAT traversal test [Was: enabling bridges on 
> > NATed clients]
> > User-Agent: KMail/1.12.4 (Linux/2.6.31-14-generic; KDE/4.3.5; i686; ; )
> > Reply-To: address@hidden
> >
> > Dear all,
> >
> > In order to more thoroughly answer sird's question (for GNUnet, possibly for 
> > Tor and generally for anyone interested in P2P), a group of people (including 
> > Andreas Mueller, Samy Kamkar, Nate Evans and myself) would like your help.
> >     
>
> I had some thoughts about building a library for applications which do fancy
> things with the underlying network. Primarily, because I have a project which
> will likely be more useful, if applications take proper use of it. (see
> http://michaelblizek.twilightparadox.com/projects/cor/internet_exit.html )
>
> To my mind such a library should do *way* more than NAT transversal, if it is
> supposed to be "generic":
> - Provide OS abstractions
>   

I very much want a library like this.  I need it for my own applications 
and there are many reasons it would be useful.
Ideally, it would work such that any application that implements the 
"overlay network" would support communication with any other application 
that is in the same set and adheres to the same rules.  This implies 
some way to filter for allowed and not allowed applications, uses, 
certified code, trusted parties, etc.  Ideally, all of the firewall, QoS 
(Netem), etc. logic of Linux should be available.

The library should have support for OS abstractions, but it also should 
segregate that into a module or plugin that can be replaced so that it 
can fit into any environment: Qt-based apps, etc.

> - Provide socks proxy (which BTW have a way for opening ports
>   on the other side) abstractions
> - Use its own configuration file, if the application does not override things
>   at runtime. This way, the user can configure everything even if the
>   application does not provide ways to set certain parameters. Also, an
>   "application name" parameter should be passed to the lib, so that the config
>   file could contain different parameters for each application.
>   

This should be flexible and plugin or otherwise replaceable.

> - Maybe provide some addressing abstractions, so that applications can run
>   within IPv4/IPv6/.onion/... networks without changes.
>   

The presence/IM networks give a great model for this, as does RabbitMQ: 
endpoints have unique names, perhaps with sub-resources (Jabber) that 
are logical and can have their own semantics, authentication, 
encryption, communications model, etc.  Pub/sub and virtual pipe (with 
channels preferably (BEEP, AMQP, etc.)) should both be supported.

> - Provide ways to set things like IP_TOS, so that external shapers are easier
>   to setup.
> - Make sure that e.g. TCP_CORK is always set when possible.
> - Do throttling the proper way and *not* by "usleep", but by setting
>   TCP_CONGESTION or smaller tcp window sizes if possible. "usleep" should only
>   be the last resort. When using usleep or smaller tcp window sizes, the lib
>   should be able to figure the proper parameters out by itself e.g. by pinging
>   a fixed IP and looking at the response times when the net is under load.
>   

Rate-based end-to-end flow control with bandwidth estimation, usually 
needed at both the communications and application level, is the way to 
solve most related problems.  I can detail and provide references.  It 
turns out not to be very hard to implement in most cases.  Recently, we 
called this "pro-active flow control" to try to distinguish it from 
simple window-based flow control.

> - Provide a way for the application to tell the library which connections are
>   important and which connections need throughput or low latency.
>   

Yes, need a standard way to provide and propagate these hints.  Also, 
need to provide visibility of preferences of routers (i.e. intermediate 
systems) along the pipeline if possible.  This allows things like 
bandwidth throttling to be visible, enforced, and accounted for (perhaps 
by opening additional channels).

>         -Michi
>   

sdw