Re: Dybvig's module system and interactivity

[Mikael Djurfeldt]

[This response may sound a bit agitated, but it's all meant quite
 friendly.]

Dirk Herrmann <address@hidden> writes:

> My position on Dybvig's module system is:  We should not use it.  Maybe
> we can figure out something that removes its problems, but at least
> now I don't see any solution.  I will try to give a better explanation of
> what I think are the problems with that system.

And I actually don't think it has any big problems.  If we simply took
it verbatim and used it together with a single top-level environment,
I think we would have a truly beautiful, self-consistent, and
extremely expressive system.  I would be quite happy with that.

*But*, I think there are possibilities to make it even more useful and
easy to work with.

I think the problems you bring up aren't problems but rather strengths
in the system.  Dybvig's module system is actually not the usual thing
we mean by a module system, but rather a tool to control and abstract
over scope.  It's an intuitive and extremely powerful low-level
language mechanism which adds substantially to the expressiveness of
the language.

It's expressive enough that one can quite beautifully define a more
conventional high-level module system as a set of macro definitions.
I envision that Guile modules will have a form of this high-level
system as module header.

> Dybvigs module system is based on the "import" primitive.  This primitive
> accepts one module as parameter and will add all exported bindings from
> that module into the current _local_ environment, thereby potentially
> shadowing other bindings.
> 
> Let's assume the following code:
> 
>  1: (let ((x foo))
>  2:   (import M)
>  3:   x)
> 
> The value x in line 3 might refer to x in line 1, but if M exports a
> binding to x, that binding will take precedence.  In other words, without
> knowing the exact set of bindings that are exported by M, it is not clear
> >From that code, to what x refers.  Thus, Dybvig requires that the set of
> exported bindings must be known at the time when the above code is
> compiled.
>
> I don't know what others think, but I would not want to have to check the
> list of exports every time I import a module.  Further, I would not want
> to do it again when I receive a new version of some module:  If the
> implementor of M thinks that adding a binding for x in the next version 
> of M is a good idea, this will break my code.  Worse:  It will break
> silently, since the code is syntactically correct before and after that
> change to M.
 
I still don't know what you experience as a problem here.  You seem to
be talking about the user level.  If so, my reply is:

If you are worried about shadowing bindings, then don't place the
import form in a local lexical scope.  The fact that you *can* place
it in a local scope is a great strength of the system.  I don't see
how this can be used as an example of a problem of this module system,
since it isn't even *possible* to do it in other module systems.  You
definitely don't have to place the import form in a local scope.

BTW, your example above is not correct.  Hygiene of binding reference
makes it impossible for module M to shadow x.  The reason why the
import of module M could influence the expressions following it in
Dybvig's example is that both the module definition and the following
expressions are within the same scope.

Also, why is there more of a need to check the export list of a module
in Dybvig's system than in any other module system?


On the implementation level it is true that one needs to know the
syntactic bindings of the modules one imports.  This is solved by
using the pre-compiled information as I mentioned earlier.

> It is stated as a big benefit of Dybvig's system that it can be easily
> compiled, because the code (or rather: the set of exported bindings) of
> the modules and the code of exported macros are known.

Isn't it nice that all module system machinery is macro-expanded away,
so that everything which is left is (potentially a subset of) R5RS
Scheme?

In other module systems, the border between modules might prevent
optimizations and inlining over module boundaries.

> But this is not surprising: _Any_ system that imports other modules
> could make use of that knowledge.

But they don't.  :)

> It is IMO a big disadvantage that Dybvig's system requires to see
> the module code, just to get the bindings even in local scopes
> right, and even if the imported module does not define a single
> macro.

The system doesn't need to see the code of other modules.  It's enough
to see the interface and the transformers.  As stated earlier, for
separate compilation such information can, for example, be stored in
object files.  In the interactive system, there's no problem, since
the transformers will be easily accessible.

> OK, so if using import within a local scope is a bad thing, why not just
> restrict its use to the top-level?  Sure, you could do that.  However, in
> this case all the examples from Dybvig's paper (the implementation of
> "from", "define-alias", ...) have to be implemented differently, because
> their implementation depends on using "import" in a local scope.  In other
> words, we would have to implement all interesting primitives for module
> handling as if we would be starting to write a new module system on our
> own anyway.

What are you saying?  It seems to me that the core of this argument
is: Dybvig's system has these extra features not found in other module
systems.  If we use them, we run into trouble.  But if we don't use
them we don't have all these nice extra features.  Therefore, Dybvig's
system has a problem.

???

:)

Mikael