Re: continuation efficiency

[Thomas Bushnell, BSG]

Marius Vollmer <address@hidden> writes:

> Suppose you have two modules: one provides `with-remote-port', the
> other uses it.  The procedure `with-remote-port' will open a new
> network connection to some server and wrap it with a port.  This port
> is then used by other code.  `with-remote-port' knows that the network
> connection is expensive and ties up significant resources on the
> server, and it would therefore like to close it reliably, and as early
> as possible.

So we have a primitive that creates a resource which is not managed by
GC.  To bring the absurdity into sharper focus, I'll call this
"magic-cons".

A magic-cons is just like a cons (and is turned by a procedure called
"magic-cons").  The only difference is that a magic-cons is a builtin
memory leak; it consumes resources, and we are going to order the
programmer to track all uses of magic-cons and call
(delete-magic-cons) when the dynamic extent of the magic-cons has
ended.  

But this has a problem: by adding a primitive to the language which
provides a non-gced resource, we now have to give the programmer
explicit additional access to the dynamic extent of objects.

Now all this hair the programmer *DOES NOT WANT* to deal with.  If the
primitive does its job right, then the programmer does not have to
deal with it.

Note that it is not possible for the programmer to create such an
object like magic-cons which requires explicit deallocation and access
to information about dynamic extents.  It only happens because of a
broken primitive.

Which means, right off the bat, that if the language implementor is
ever asked by a user to implement {call/ec,escape-wind}, he should
look closely at what the user is doing.  He will find that the user is
using a broken primitive, and he should fix the primitive.  The user
will be much happier, because the user didn't want to have to manage
his own memory anyway.

> All it wants is a reliable notification that the dynamic extent of
> `with-remote-port' has ended.

Translating, the user only wants a reliable notification of when the
dynamic extent of a magic-cons has ended so that he can call
delete-magic-cons.  Better for the GC system to do it!

> How should `with-remote-port' be written?  This version
> 
>     (define (with-remote-port proc)
>       (let ((port (make-remote-port)))
>         (proc port)
>         (close-remote-port port)))
> 
> falls obviously short because when control leaves PROC non-locally,
> PORT is not closed.  

This is the classic Lisp example, but it's actually wrong-headed.

In the real world, we are implementing a network protocol.  Closing
the port is not something we do "when we are finished with it"; it's a
protocol operation, and it's illegal to close it when the transaction
isn't done.  Closing is like writing "RCTP to: <address@hidden>" on the
port.  You can't just do it willy-nilly, you have to do it once, at
the right time.

The Lisp example wraps an unwind-protect around the let, and claims
it's now better.  Well, no.  It no longer leakes the network port, but
it's not any better: it violates the protocol even so.  Instead, the
correct thing is for the *proc* routine to close the port *correctly*
upon an error exit, which is something that you only know how to do in
general if you know what the protocol is being spoken.

So lets look a little more under the hood of how I think this should
be done.  I'm still a little unsure, so bear with me.

First, the *only* resource that is "expensive" is whatever remote
resources we are holding on to.  But in general, we don't hold on to
resources so that we can "maybe" reuse them: we only hold on to them
when we actually know we want to reuse them.

The danger supposedly avoided by using unwind-protect (in Lisp) or
dynamic-wind (in Scheme) around the call to proc is that an error will
happen, we'll totally abandon proc, and then the port will be "left
open" forever and needlessly.  That's a fake worry.  A correct system
will gc the port object as soon as this happens, if indeed, we have
totally abandoned proc.

Now that closes the port, freeing the remote resources.  But it
violates the protocol.  If you want to honor the protocol, then you
need to add a finalizers option to the procedure make-remote-port.
Again, this is all only necessary because make-remote-port is being
specified ahead of time as being broken: as being something that holds
magic non-gc'd resources.

Now that's the party line I want to sell.  

As an addendum, I want to note that it *does* still have a problem.
If we write a system entirely in Scheme, we still have network
connections, and those connections use remote resources, and we want a
way to have a remote connection closed when a PCB gets gc'd.  Note
that this is *not* a problem in an embedded implementation, for which
the network connection is implemented as a primitive; that primitive
should just know how to GC its objects the way every single other
primitive is expected to.

How might that happen?  One way would be through a general
finalization mechanism in the GC system.  I have an inkling there are
other solutions too.

So, at the end of this long mail, let me refine my challenge to
Marius:

Can you give an example where call/ec and escape-protect are needed
where you are *not* positing in advance that the gc system is broken?

Thomas