Re: vm status update

[Neil Jerram]

2008/12/26 Andy Wingo <address@hidden>:
> Happy St. Stephen's Day, hackers of the good hack!

More eating the good pie than hacking the good hack right now!

> I just landed a few patches on the vm branch that integrate backtrace
> handling between the interpreter and the VM. `save-stack' saves the VM
> stack and the interpreter stack, properly interleaved, as your
> computation bounces back and forth between the interpreter and the VM.
>
> This also means that we now get VM backtraces from scm_backtrace() and
> friends.

That sounds great.

> There are still some hiccups, but this was one of the last things I
> needed to get done before merging VM to master. I need to finish
> updating the documentation, fix the tracing infrastructure to be like
> the traps infrastructure (it already is, mostly), and we'll be done.

Nice.  Regarding the merge to master, though,

- I think that would imply that the VM is included in the next release
series (1.10.x or 2.0.x); is that your intention?  (I have no
objection!)

- if yes to that, even so it might help us to hold off on the merge
for now, until we've finished reviewing everything else in
branch_release-1-8..master, and moving _out_ anything that should be
in the next release series.

Now the last point makes it sound as though there's actually a plan
here, and I'm afraid there isn't really.  But I've recently been
looking at the branch_release-1-8..master diffs, and reverting some
that shouldn't have been there, and I think Ludovic may have been
doing a little of that too.  I don't think it will take very long for
us to complete that, and hence to be confident that everything in
master is ready for release.

Would that be OK?

> Finally, as part of the documentation work, I wrote up a history of
> Guile. I'm including it in this mail for comment, but here's the link if
> you want to read it that way:
>
>  
> http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=doc/ref/history.texi;hb=vm
>
> I still need to fold in some feedback from Ludovic, so consider the
> document a draft at this point.
>
> Cheers,
>
> Andy
>
> * begin history.texi:
>
> @c -*-texinfo-*-
> @c This is part of the GNU Guile Reference Manual.
> @c Copyright (C)  2008
> @c   Free Software Foundation, Inc.
> @c See the file guile.texi for copying conditions.
>
> @node History
> @section A Brief History of Guile
>
> Guile is an artifact of historical processes, both as code and as a
> community of hackers. It is sometimes useful to know this history when
> hacking the source code, to know about past decisions and future
> directions.
>
> Of course, the real history of Guile is written by the hackers hacking
> and not the writers writing, so we round up the section with a note on
> current status and future directions.
>
> @menu
> * The Emacs Thesis::
> * Early Days::
> * A Scheme of Many Maintainers::
> * A Timeline of Selected Guile Releases::
> * Status::
> @end menu
>
> @node The Emacs Thesis
> @subsection The Emacs Thesis
>
> The story of Guile is the story of bringing the development experience
> of Emacs to the mass of programs on a GNU system.
>
> Emacs, when it was first created in its GNU form in 1984, was a new
> take on the problem of ``how to make a program''. The Emacs thesis is
> that it is delightful to create composite programs based on an
> orthogonal kernel written in a low-level language together with a
> powerful, high-level extension language.
>
> Extension languages foster extensible programs, programs which adapt
> readily to different users and to changing times. Proof of this can be
> seen in Emacs' current and continued existence, spanning more than a
> quarter-century.
>
> Besides providing for modification of a program by others, extension
> languages are good for /intension/ as well. Programs built in ``the
> Emacs way'' are pleasurable and easy for their authors to flesh out
> with the features that they need.

I'm afraid I don't understand this last paragraph!  i.e. what you mean
by "intension" or "built in the Emacs way".

> After the Emacs experience was appreciated more widely, a number of
> hackers started to consider how to spread this experience to the rest
> of the GNU system. It was clear that the easiest way to Emacsify a
> program would be to embed a shared language implementation into it.
>
> @node Early Days
> @subsection Early Days
>
> Tom Lord was the first to fully concentrate his efforts on an
> embeddable language runtime, which he named ``GEL'', the GNU Extension
> Language.
>
> GEL was the product of converting SCM, Aubrey Jaffer's implementation
> of Scheme, into something more appropriate to embedding as a library.
> (SCM was itself based on an implementation by George Carrette, SIOD).

((((Last full stop should be inside the parentheses.))))

> Lord managed to convince Richard Stallman to dub GEL the official
> extension language for the GNU project. It was a natural fit, given
> that Scheme was a cleaner, more modern Lisp than Emacs Lisp. Part of
> the argument was that eventually when GEL became more capable, it
> could gain the ability to execute other languages, especially Emacs
> Lisp.
>
> Due to a naming conflict with another programming language, Jim Blandy
> suggested a new name for GEL: ``Guile''. Besides being a recursive
> acroymn, ``Guile'' craftily follows the naming of its ancestors,
> ``Planner'', ``Conniver'', and ``Schemer''. (The latter was truncated
> to ``Scheme'' due to a 6-character file name limit on an old operating
> system.) Finally, ``Guile'' suggests ``guy-ell'', or ``Guy L.
> Steele'', who, together with Gerald Sussman, originally discovered
> Scheme.

(I never heard that one before! (i.e. the last sentence) Interesting.)

> Around the same time that Guile (then GEL) was readying itself for
> public release, another extension language was gaining in popularity,
> Tcl. Many developers found advantages in Tcl because of its shell-like
> syntax and its well-developed graphical widgets library, Tk. Also, at
> the time there was a large marketing push promoting Tcl as a
> ``universal extension language''.
>
> Richard Stallman, as the primary author of GNU Emacs, had a particular
> vision of what extension languages should be, and Tcl did not seem to
> him to be as capable as Emacs Lisp. He posted a criticism to the
> comp.lang.tcl newsgroup, sparking one of the internet's legendary
> flamewars. As part of these discussions, retrospectively dubbed the
> ``Tcl Wars'', he announced the Free Software Foundation's intent to
> promote Guile as the extension language for the GNU project.
>
> It is a common misconception that Guile was created as a reaction to
> Tcl. While it is true that the public announcement of Guile happened
> at the same time as the ``Tcl wars'', Guile was created out of a
> condition that existed outside the polemic. Indeed, the need for a
> powerful language to bridge the gap between extension of existing
> applications and a more fully dynamic programming environment is still
> with us today.

That's really nicely put!

> @node A Scheme of Many Maintainers
> @subsection A Scheme of Many Mantainers
>
> Surveying the field, it seems that Scheme implementations correspond
> with their maintainers on an N-to-1 relationship. That is to say, that
> those people that implement Schemes might do so on a number of
> occasions, but that the lifetime of a given Scheme is tied to the
> maintainership of one individual.
>
> Guile is atypical in this regard.
>
> Tom Lord maintaned Guile for its first year and a half or so,
> corresponding to the end of 1994 through the middle of 1996. The
> releases made in this time constitute an arc from SCM as a standalone
> program to Guile as a reusable, embeddable library, but passing
> through a explosion of features: embedded Tcl and Tk, a toolchain for
> compiling and disassembling Java, addition of a C-like syntax,
> creation of a module system, and a start at a rich POSIX interface.
>
> Only some of those features remain in Guile. There were ongoing
> tensions between providing a small, embeddable language, and one which
> had all of the features (e.g. a graphical toolkit) that a modern Emacs
> might need. In the end, as Guile gained in uptake, the development
> team decided to focus on depth, documentation and orthogonality rather
> than on breadth. This has been the focus of Guile ever since, although
> there is a wide range of third-party libraries for Guile.
>
> Jim Blandy presided over that period of stabilization, in the three
> years until the end of 1999, when he too moved on to other projects.
> Since then, Guile has had a group maintainership. The first group was
> Maciej Stachowiak, Mikael Djurfeldt, and Marius Vollmer, with Vollmer
> staying on the longest. By late 2007, Vollmer had mostly moved on to
> other things, so Neil Jerram and Ludovic Courtès stepped up to take on
> the primary maintenance responsibility.
>
> Of course, a large part of the actual work on Guile has come from
> other contributors too numerous to mention, but without whom the world
> would be a poorer place.
>
> @node A Timeline of Selected Guile Releases
> @subsection A Timeline of Selected Guile Releases
>
> @table @asis
> @item guile-i --- 4 February 1995
> SCM, turned into a library.
>
> @item guile-ii --- 6 April 1995
> A low-level module system was added. Tcl/Tk support was added,
> allowing extension of Scheme by Tcl or vice versa. POSIX support was
> improved, and there was an experimental stab at Java integration.
>
> @item guile-iii --- 18 August 1995
> The C-like syntax, ctax, was improved, but mostly this release
> featured a start at the task of breaking Guile into pieces.
>
> @item 1.0 --- 5 January 1997
> @code{#f} was distinguished from @code{'()}. Green threads were added.
> Source-level debugging became more useful, and programmer's and user's
> manuals were begun. The module system gained a high-level interface,
> which is still used today in more or less the same form.
>
> @item 1.1 --- 16 May 1997
> @itemx 1.2 --- 24 June 1997
> Support for Tcl/Tk and ctax were split off as separate packages, and
> have remained there since. Guile became more compatible with SCSH, and
> more useful as a UNIX scripting language. Libguile can now be built as
> a shared library, and third-party extensions written in C became
> loadable via dynamic linking.
>
> @item 1.3.0 --- 19 October 1998
> Command-line editing became much more pleasant through the use of the
> readline library. The initial support for internationalization via
> multi-byte strings was removed, and has yet to be added back, though
> UTF-8 hacks are common. Modules gained the ability to have custom
> expanders, which is still used for syntax-case macros

(and the preliminary Emacs Lisp support)

>. Ports have
> better support for file descriptors, and fluids were added.
>
> @item 1.3.2 --- 20 August 1999
> @itemx 1.3.4 --- 25 September 1999
> @itemx 1.4 --- 21 June 2000
> A long list of lispy features were added: hooks, Common Lisp's
> @code{format}, optional and keyword procedure arguments,
> @code{getopt-long}, sorting, random numbers, and many other fixes and
> enhancements. Guile now has an interactive debugger, interactive help,
> and gives better backtraces.
>
> @item 1.6 --- 6 September 2002
> Guile gained support for the R5RS standard, and added a number of SRFI
> modules. The module system was expanded with programmatic support for
> identifier selection and renaming. The GOOPS object system was merged
> into Guile core.
>
> @item 1.8 --- 20 February 2006
> Guile's arbitrary-precision arithmetic switched to use the GMP
> library, and added support for exact rationals. Green threads were
> removed in favor of POSIX threads, providing true multiprocessing.
> Gettext support was added, and Guile's C API was cleaned up and
> orthogonalized in a massive way.
>
> @item 2.0 --- thus far, only unstable snapshots available
> A virtual machine was added to Guile, along with the associated
> compiler and toolchain. Support for locales was added. Running Guile
> instances became controllable and debuggable from within Emacs, via
> GDS. GDS was backported to 1.8.5. An SRFI-compatible interface to
> multithreading was added, including thread cancellation.
> @end table

OK, that answers one of my questions above, and persuasively so.

> @node Status
> @subsection Status, or: Your Help Needed
>
> Guile has achieved much of what it set out to achieve, but there is
> much remaining to do.
>
> There is still the old problem of bringing existing applications into
> a more Emacs-like experience. Guile has had some successes in this
> respect, but still most applications in the GNU system are without
> Guile integration.
>
> Getting Guile to those applications takes an investment, the
> ``hacktivation energy'' needed to wire Guile into a program that only
> pays off once it is good enough to enable new kinds of behavior. This
> would be a great way for new hackers to contribute: take an
> application that you use and that you know well, think of something
> that it can't yet do, and figure out a way to integrate Guile and
> implement that task in Guile.
>
> With time, perhaps this exposure can reverse itself, whereby programs
> can run under Guile instead of vice versa, eventually resulting in the
> Emacsification of the entire GNU system. Indeed, this is the reason
> for the naming of the many Guile modules that live in the @code{ice-9}
> namespace, a nod to the fictional substance in Kurt Vonnegut's
> novel, Cat's Cradle, capable of acting as a seed crystal to
> crystallize the mass of software.
>
> Implicit to this whole discussion is the idea that dynamic languages
> are somehow better than languages like C. While languages like C have
> their place, Guile's take on this question is that yes, Scheme is more
> expressive than C, and more fun to write. This realization carries an
> imperative with it to write as much code in Scheme as possible rather
> than in other languages.
>
> These days it is possible to write extensible applications almost
> entirely from high-level languages, through byte-code and native
> compilation, speed gains in the underlying hardware, and foreign call
> interfaces in the high-level language. Smalltalk systems are like
> this, as are Common Lisp-based systems. While there already are a
> number of pure-Guile applications out there, users still need to drop
> down to C for some tasks: interfacing to system libraries that don't
> have prebuilt Guile interfaces, and for some tasks requiring high
> performance.
>
> The addition of the virtual machine in Guile 2.0, together with the
> compiler infrastructure, should go a long way to addressing the speed
> issues. But there is much optimization to be done. Interested
> contributors will find lots of delightful low-hanging fruit, from
> simple profile-driven optimization to hacking a just-in-time compiler
> from VM bytecode to native code.
>
> Still, even with an all-Guile application, sometimes you want to
> provide an opportunity for users to extend your program from a
> language with a syntax that is closer to C, or to Python. Another
> interesting idea to consider is compiling e.g. Python to Guile. It's
> not that far-fetched of an idea: see for example IronPython or JRuby.
>
> And then there's Emacs itself. Though there is a somewhat-working
> Emacs Lisp translator for Guile, it cannot yet execute all of Emacs
> Lisp. A serious integration of Guile with Emacs would replace the
> Elisp virtual machine with Guile, and provide the necessary C shims so
> that Guile could emulate Emacs' C API. This would give lots of
> exciting things to Emacs: native threads, a real object system, more
> sophisticated types, cleaner syntax, and access to all of the Guile
> extensions.
>
> Finally, there is another axis of crystallization, the axis between
> different Scheme implementations. Guile does not yet support the
> latest Scheme standard, R6RS, and should do so. Like all standards,
> R6RS is imperfect, but supporting it will allow more code to run on
> Guile without modification, and will allow Guile hackers to produce
> code compatible with other schemes. Help in this regard would be much
> appreciated.

Fantastic.  I completely take back my initial concerns about this
history.  Thank you very much for writing it!

Regarding the detail: everything that you have written is consistent
with the history that I know of, but in several places is more
detailed than I knew.  So I can't verify all of those details, but I
have no reason to suspect that any of them are wrong.

Regards,
        Neil