Re: Compilers for Guile (related to roadmap and goals)

[Rob Browning]

Tanel Tammet <address@hidden> writes:

> 5) I find this very disturbing.
> IMHO, one of the main goals for Guile (which is NOT an early 
> beta system) should be to guarantee out-of-box compilability
> on most distros. It is clear that problems like this push 
> potential users away from Guile.

Agreed -- guile has had release problems in the past -- we're trying
to fix those.

> 6) Licencing issue: Guile is licenced under GPL PLUS EXCEPTION which
> allows one to link against it without the end product being
> GPL-d. IMHO it is an extremely liberal and user-friendly
> licence. However, when you donwload Guile, look into the manual,
> look into COPYING, it is nowhere to be seen that there is such an
> exception (which, for practical purposes, completely removes the
> standard GPL obstacles). The only place the exception is stated, is
> in the .c and .h file headers.  Users won't find the exception! They
> will be under the false impression that any link against Guile is
> GPL-d. IMHO it would be important to clearly state the exception in
> the manual and in the top-level directory.

Absolutely -- it should in COPYING -- I'll look in to that.  It is in
/usr/share/doc/guile1.4/copyright on Debian systems.

> The first and obvious thing for me would be taking up 
> the existing Guile port of Hobbit, and:
>
> 0) find if other people are working on compilers for Guile.

You should probably read Marius (and my) recent mails regarding
compilation and Guile.  Marius has some farily clear ideas about what
he wants to do next.  Also relevant would be the discussions from last
year regarding Keisuke's VM and Marius' work with the Lightning JIT.
If you can't find these, I'll try and help -- also if anyone else has
handy pointer(s), please holler.

> 1) check if it is still working together with Guile

It's not.  I've worked with hobbit and guile CVS quite a bit (I think
I sent you some messages, but I don't know that you ever got them...),
and after that experience and talking with Marius both on and off the
list, it seemed like Guile probably needed some cleaning up before it
was time to work on compilation in earnest again.  One important
stumbling block at the moment is syntax-case, or more broadly guile's
macro system, especially when considered in the context of "what
happens when".  i.e. it seems like we need a clearer distinction
between "load time", "compile time", and "eval time".  When are top
level forms evaluated?  What happens if you execute code with side
effects when defining a macro, etc.

> 2) if not, try to fix it

See Marius recent repost to -devel of a private message to me (for his
rough ideas).

> Some comments and questions regarding compilation follow.
>
> - Is speed necessary?

*YES*.

>   Obviously, many Guile developers do not think speed is important
>   (otherwise Guile would be at least as fast as SCM and would have a
>   compiler).

You should be careful with generalizations like that.  Guile has had
quite a few of its major developers come and go, and AFAICT has been
in somewhat a "lull" for the past couple of years.  I'm relatively
new, and am only now having the time to get up to speed on some of the
more critical guile internals, bits that would likely be important in
any work relating to compilation.

If we could manage it I'd love to see guile designed so that depending
on the restrictions you're willing to impose on your code, you can use
guile for the entire range of your computing needs (performance-wise).
I.e. I'd love to see guile have a good very friendly interpreter with
extensive debugging support, but also have a switch you can flip that
will compile a file with a bunch of restrictions and get speeds to
rival stalin.  Now perhaps we'll never get that far, but I certainly
wouldn't mind if we did :>

> 2) Because of this, scalable web apps are often built in
>    Java or C: in a serious setting (many users, heavy loads,
>    nontrivial programs) it DOES matter if your program becomes
>    100 times faster (or slower).

GACK -- my experiences with java have not suggested to me that java's
any speed daemon, but I try  not to use java any more than I have to.

[ rest of attempt to preach to the converted omitted :> ]

> * byte codes
> * direct machine code

Marius is also quite interested in GNU Lightning, and I'm somewhat
intrigued myself.

> -----------
> New developments have made it possible to create very fast byte code
> interpreters.  We are willing to replace the current Guile evaluator
> when a suitable candidate byte code interpreters turns up.
> ------------

I believe this is a pretty old entry and doesn't realy reflect some of
the recent discussions -- Thien-Thi's definitely right that we need to
start working more on some of these meta-level docs.

> I am an avid supporter of direct machine code, not byte code,
> for scheme. Why so:

While I agree with your arguments somewhat, I've seen some reasonably
fast byte-code systems, and in cases where you can tune them with
suitably large primops etc. you should be able to get some benefits
from interactions with the memory heirarchy (cache/locality/etc).

However, I'm *personally* a little more intrigued by JIT or direct C
compilation, while still being open to the possibilty of a good VM.

> 5) If we cannot create a good JIT, then byte codes will be

You seem to equate JIT with generating java byte codes.  I'm not sure
that's our only approach.  I suppose that depends in part on what the
likely future prospects for Lightning are.

One other thing I've been wondering about is whether or not we could
use gcc as an on-demand compiler -- i.e. what if we could spit code
out to gcc, compile it, and dynamic-link it back in on demand.  This
wouldn't be something you'd do every time you launch the app, but it
could be one approach to compilation.  I've been meaning to see what
libgcc is all about, but haven't had time yet...

> 1) For me personally it would be easy to base the work on Hobbit: I
> know it, it works nicely with SCM, is now included in the standard
> SCM distro, etc.  It has been ported to Guile once (although I
> gather from the discussion and other information, that this port is
> no longer supported for Guile).

AFAICT Hobbit support wasn't dropped intentionally; it looks like
hobbit wasn't ever updated to accomodate the addition of the
syntax-case subsystem, which itself is kind of a "mini compiler".

> 2) There are many scheme-to-c compilers.

I'd even wondered if there were some way we could borrow methods/code
from stalin, and allow specified sections of code to be compiled under
*very* strict assumptions to extremely fast code.

>      Many interpret-time special operations cannot be compiled

Right I think if/when we need to, it's OK to just assert things like
"in some cases if you call eval then you may just lose
(performance-wise)".

>      When you can do that, then of course you can code most of your
>      program in scheme instead, and use a scheme-to-c compiler to
>      produce the C code you want to integrate with Guile. This may
>      be a tremendous help, even if you cannot compile full Guile
>      (you would not hand-code force, delay, hygienic macros etc in c
>      anyway :)

And we may be able to help compilation with declarations or perhaps
even a special primitive language.  i.e. for intense integer code, we
could have special ops like fixnum+, fixnum-, etc. which if you use in
a block, with no other "contaminating" math code, will be recognized
by the compiler and then the whole block will be turned into a *very*
efficient C translation.

>    - code relying on fast (proper) implementation of tailrecursion
>      for mutually tail recursive functions. Hobbit (and Bigloo, for
>      example) only recognise tail recursion inside a function, not
>      mutual tail tail recursion between different functions.

This brings up one other question I've been wondering about -- anyone
know what ever happened to the possiblity of getting tail-recursion
(as a special option) into gcc?

> And now, the final question after the long story: is anybody from
> the Guile community right now hacking Hobbit or some other
> scheme-to-c compiler for Guile?

No.

> Has anybody done that last year?

Yes :> (I messed with Hobbit and toyed a bit with Marius' Lightning
work and Keisuke's VM).

> If yes, please take contact so that we could perhaps coordinate
> future work.

I think you may have misinterpreted the mood here.  There are a number
of people here, a number of people I work with, quite a few Gnucash
developers, and I suspect even more people on guile-user who would be
very interested in compilation for guile.  In addition there have been
several somewhat lengthy discussions about some of this both on and
off the list over the past year or two.

However, right now, during my Guile-allocated time I'm primarily
focused on getting 1.6.1 out.  That should happen *VERY SOON* or I'm
going to get increasingly cranky on the lists :>

If it were up to me, I'd like us to focus on the release and try as
much as possible to wait until after that to discuss most of the
really cool future issues like compilation, internationalization of
strings, what we might or might not want to do with the VM, elisp,
etc.

In any case thanks for your comments, and good to hear from you.

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD