Re: goops and memoization

[Mikael Djurfeldt]

Dirk Herrmann <address@hidden> writes:

>>     Mikael> It seems to me that what you need to do is to run the
>>     Mikael> tail of the cmethod (BODY-FORM1 ...) through your
>>     Mikael> memoizer.  That should fix it.

[...]

> Simple re-memoization is also not possible, except you use special
> tricks, similar to using local-eval.

I don't understand why this is so complicated.  I haven't seen your
memoizer, but doesn't it consist of recursively called function(s)
which pass an argument representing the local environment, just as
eval and the unmemoizer?  As you know, a cmethod has the local
environment as a component.  Can't you use that?  You could then
implement a primitive which can memoize a cmethod, and you could call
it at the end of compile-method just as Neil suggests.

>> - You could just call eval instead of memoizing and goto
>>   nontoplevel_begin, but that wouldn't be tail-recursive.  I wonder if
>>   tail recursion is important here?
>
> I can't tell.

It is important.  GOOPS methods need to handle tail-recursion nicely.

> procedure-source returns post-transformation code.  But it is still a
> hack.  Further, it is not a good idea to rely on the fact that is returns
> post-transformation code:  From a debugging perspective, it might be
> better to return the original code.  It is IMO bad style to rely on the
> behaviour of procedure-source.  It was a short term hack, as the comments
> in goops indicate.

I don't agree that this in particular is a hack.  compile-method needs
to work on some representation of the logic of a method.  I think the
source is an excellent representation for the logic of the method, and
we have a primitive in the Guile API which provides the source:
procedure-source.  I actually think working on the memoized
representation is more of a hack, but that might be motivated anyway
for reasons of efficiency.

Then we can argue about the inner workings and the exact result of
procedure-source, but that is a different issue.  Maybe we should
finally store the original source somewhere, but currently there is
not much difference between unmemoized source and the original.  The
only difference is that the (irrelevant) distinction between let and
let* for a single binding is lost.

Also, note that for GOOPS, it doesn't matter much how well the source
resembles the original source as long as the semantics is intact.

>> - Quite a few places in the GOOPS code use local-eval.  Does
>>   local-eval still include memoization (and syntax transformation?) in
>>   your codebase?
>
> Yes, but IMO local eval is also quite hackish.  It is placed in debug.c
> which indicates to me that it was planned as a debugging aid.  It should
> probably be avoided where possible.  Maybe it is just a bad idea anyway.

It is placed in debug.c for obscure historic reasons.  But you're
right that we may want to remove local-eval in the future.  (BTW, note
that some scheme interpreters, like MIT scheme does support evaluation
in local environments.)

However, as long as the interpreter is organized the way it is now, I
see no reason why we can't use it internally if that gives
performance.  It's pretty easy to change when we want to do that.

> I think the whole concept of unmemoization and re-memoization is broken.
> IMO the optimizations that are done by goops should either be performed on
> scheme code or some close-to-scheme intermediate representation before
> memoization, or if they are based on already memoized code, they should
> work on the memoized code itself.  Mixing different stages as it is
> currently done introduces a lot of dependencies between different parts of
> guile.  This makes maintaining guile quite hard - our current discussion
> is already an example of the problem.

As I say above, I think one can look upon it differently: We have
procedure-source in our API.  That call will always be supported, so
we can rely on it.  Then it's completely OK to do optimizations and
then memoize, just as we'd memoize novel source.  But then, again, if
we want method compilation to work fast, we might want to work in the
memoized source, with the disadvantage that we'd be bound to do all
manipulations on the Scheme level and couldn't include method
compilation in the MOP.

> It would be great if you could look into places in goops (or, maybe
> even guile in general) where such interactions occur.  If I had a
> list of these places, I could then try to remove the optimizing code
> from the scheme level and re-code it in C, where it would be
> possible to perform the optimizations directly on the memoized code.

I'm not sure what you mean by "interactions", but as far as I remember
right now, the *only* reason why GOOPS doesn't work with your changes
is that the output of compile-method should now be memoized.

Again, can't you just provide a primitive which can take a cmethod
(which has environment and body as components) and memoize it?  That
primitive can then by applied at just the location in compile-cmethod
where Neil suggested.

On the other hand: Sure, it would be very nice if you re-implemented
compile-method in C.  Please tell me if you have any questions.

Best regards,
Mikael