Waddell/Dybvig module system

[Neil Jerram]

>>>>> "Mikael" == Mikael Djurfeldt <address@hidden> writes:

    Mikael> Given the limited resources we have to conduct the
    Mikael> research, design and development that would be required to
    Mikael> develop a good module system of our own, and given the
    Mikael> excellent quality of Dybvig's new module system, I propose
    Mikael> that we simply take Dybvig's system and adopt it for
    Mikael> Guile:

    Mikael>   http://www.cs.indiana.edu/~dyb/papers/popl99.ps.gz

First off - we should give credit where credit is due.  Oscar
Waddell's name appears first on this paper, and Prof. Dybvig isn't
exactly short of credit already!  Hence the changed subject line.

    Mikael> We would probably make a few extensions, though.  One such
    Mikael> extension is that we'd like to handle hierarchical module
    Mikael> names which can be mapped onto a corresponding file
    Mikael> hierarchy, the way we do now.

I think it is cleaner to view our existing hierarchical module names
as a way of locating the code that is to be imported as a module.
Something like:

(use-module <hierarchical name>)
-->
(module* <interface>
  (load (module->file-name <hierarchical name>)))

The obvious problem here, though, is that <interface> has to be
specified at the point where module* or use-module is written, whereas
in our current module system <interface> is specified inside
(module->file-name <hierarchical name>).  I can't currently think of a
solution to this.

    Mikael> Probably, the nicest way to do this is to define a new set
    Mikael> of syntax handling Guile modules and write these macros in
    Mikael> the language provided by Dybvig's system.

Yes, but how? - see above.  More generally, I'm having difficulty
understanding how the translation-time Waddell/Dybvig system would fit
with aspects of the existing Guile system that seem to be more
evaluation-time/dynamic.

For example, I believe the current system uses `load' when it needs to
import a module, and caches the result of the `load' so that further
`use-module's don't cause the imported module code to be evaluated
again.  I don't understand how this kind of caching can be achieved
with Waddell/Dybvig's system.

Also, I don't see how it fits at all with the environments API, which
as far as I understand is completely dynamic (evaluation-time).

    Mikael> While Dybvig's system has many arguments speaking for it
    Mikael> as a candidate module system for Guile (e.g. that it mixes
    Mikael> very well with the syntax-case macros), the most
    Mikael> controversial point is that it might turn out that the
    Mikael> best way to implement it requires using information from
    Mikael> previously compiled modules when expanding a new module.

    Mikael> That is, when compiling (or interpreting for that matter)
    Mikael> one module, we need some macro definitions from the
    Mikael> external modules we import from.  The system would not use
    Mikael> the source code of those modules, but instead the object
    Mikael> code produced by the compilation, or some other kind of
    Mikael> file produced when expanding a module.

    Mikael> We can look upon these files as corresponding to the
    Mikael> pre-compiled headers of some C development environments.

I don't completely understand the details here, but I believe the
paper's assurances that it is only necessary to know the interface and
the transformers of the other module, and I think that sounds
acceptable.

    Mikael> The main reason why we need this is the power and
    Mikael> expressivness of Dybvig's system: Since the macro
    Mikael> expanders have the full power of the Scheme language
    Mikael> available to them, it can happen that some expanders are
    Mikael> non-deterministic---they might have different expansions
    Mikael> at different times of evaluation, which means that we
    Mikael> can't just use the expander definitions a second time to
    Mikael> deduce how to connect bindings across modules when
    Mikael> importing.

That's not what I thought the paper said at all.  The expansion
process is always deterministic.  The problem is that the meaning and
binding of an identifier in the module-importing code cannot in
general be determined until all the syntax surrounding it has been
expanded.

    Mikael> Here are a few points that I'd like us to discuss:

    Mikael> * Do we want to use Dybvig's module system?

It's funny.  On the one hand, I sort of don't like the way that this
module system confuses (extends) the definition of lexical scoping.
("How on earth am I going to describe this in the reference manual?")
On the other hand, I rather like the system for exactly the same
reason: it's elegant because that (extending the definition of lexical
scoping) is _all_ it does.  (Plus bookkeeping stuff to make sure that
macros continue to work as expected.)  On balance, I think I like the
elegance and expressive advantages more than I dislike the confusion
of lexical scope.

I'm worried about implementing it in Guile, though.  The paper speaks
strongly for elegance of design in the form of (i) a very simple core
language plus (ii) lots of syntactic abstraction.  I agree that this
is very nice.  But it seems to me that elegance of design is not so
good if it doesn't translate into elegance of implementation.  And it
is notable that Guile already is not implemented with this kind of
elegance: instead of processing a bunch of define-syntax expressions
at start up, the translator/evaluator has special case code for do,
cond, case, let etc.  So I worry that, if we tried to implement the
Waddell/Dybvig module system in Guile, we might not actually benefit
from its elegance of design, but instead end up with more spaghetti
code in the translator/evaluator.

Clearly the special cases were put there originally for performance
reasons.  Is it possible to achieve optimum performance while
following the design elegance of simple core + syntactic abstraction
in the implementation?  If yes, I think we should do that first to
check, and then implement the module system.  If no, I think that's a
strong argument against this module system.

    Mikael> * How would we like to extend it for Guile?  Specifically,
    Mikael> how should the Guile meta-modules (the new set of syntax
    Mikael> for defining Guile modules) look like?

Personally, I favour the recent extensions to define-module that allow
all the module meta-information to be specified in the define-module
expression, as in:

(define-module <name>
  :use-module ...
  :use-module ...
  :export ...
  :pure ...
  :export-syntax ...)

If it is possible to implement this syntax using the new module
system, I think we should do so.  I don't see how export and
define-public can be implemented using the proposed module system, but
then I wouldn't mind if they disappeared.

    Mikael> * Is it OK to generate expansion information in a separate
    Mikael> file?  If not, is there some way we can avoid it?  (We
    Mikael> could for example restrict the language available to the
    Mikael> expanders so that they behave deterministically.)

I think the problem here is not storing information in a separate
file; the problem is specifying how an interactive system behaves when
certain things are redefined, as discussed in another thread.

Regards,
        Neil