Re: Evolution & optimization of the module system

[Kevin Ryde]

address@hidden (Ludovic Courtès) writes:
>
> (1) has to do mainly with `module-use!' vs. `module-use-interfaces!' (as
> was discussed recently).  Namely the fact that duplicate processing is
> not always performed, depending on whether one uses `module-use!' or
> some other means to use a module.

There's nothing broken though is there, it's just a trap for the
unwary (like me for instance, wrestling with use-srfis)?

> There's still more to do to achieve (2) (notably actual documentation
> ;-))

I've put that off, because a lot looks like internals, and I'd
wondered if r6rs might offer some of its own "introspection" stuff.

> The patch solves this issue by making
> duplicate processing inescapable.  Likewise, variable lookup currently
> has two implementations (which have the same behavior, though): the C
> `module_variable ()' and the Scheme `module-variable'.  The patch leaves
> only one implementation of that.

Moving stuff into C is a good thing.  You can propose that separately
if you like, it should be able to go straight in.  (And could be a
candidate for the 1.8 branch if the performance help is noticable.)

> Although duplicates should be the exception rather than the rule[*],
> duplicate processing is pretty costly: the current `process-duplicates'
> is roughly O(N*USES),

USES is supposed to be smallish of course though.

> Likewise,
> variable lookup (e.g., in `module_variable ()') is O(USES).  I believe
> that both may have a sensible impact on startup time.

Something definitely slows down startup over 1.6.  I'd guessed it was
strings, but I failed miserably at making either gprof or
functioncheck give some call counts to prove or disprove that.

> The patch addresses this by changing the data structures used by
> modules: instead of a list of used modules, it uses a second "obarray",
> called the "import obarray", that maps symbols to the modules providing
> them.

That would increase the memory used by a module though, would it?  Big
modules imported into lots of other places would have their binding
list much copied would they?  (I'm thinking for instance of gtk and
gnome, and thinking purely selfishly since I import gtk and/or gdk
into a bunch of my modules.)

> This has several implications.  First, duplicate processing occurs the
> same way for dynamically added bindings than for "statically imported"
> bindings.  Second, it makes load-time-dependent duplicate policies such
> as `last' and `first' irrelevant (since they are inherently
> non-deterministic).

Last and first still sound pretty sensible.

> Third, it makes dynamic addition of bindings relatively costly.

Depends how often that happens I suppose.  There's no actual
documented way to do it is there? :-)  Maybe some tomfoolery with
macros and eval ...

> (2) the
> SMOB classes are added to a separate module called `(oop goops
> smob-classes)'.  Since only `(oop goops)' uses it, it is the only one
> that needs to re-process duplicates as new SMOB classes are added.

Sounds ok in principle.

> From the measurements I've made, the new version is
> around 40 times faster than the other one.

Is that in part due to moving the variable lookup to C?

> So the question is: is the `beautify-user-module!' overhead compensated
> by the variable lookup and duplicate processing gains?

I'd be inclined to say probably not, but that maybe there's another
way.

Duplicates is trying to find the intersection between one set of
symbols (from one module) and some reasonably smallish number N of
other sets (from other modules) is it?  I wonder if there's some way
to find that more efficiently.

I guess the `warn' in the default duplicates handling makes it
necessary to find all the duplicates.  Nice enough feature, but
without it there'd be no need to check any duplicates at all, or only
check against the few `#:replace' lists.

Maybe the worst afflicted programs should be advised to turn it off,
or even turn it off globally when confident there's no accidental
clashes.  And for a start I guess we don't need any duplicates work in
the core, at least up until reaching the `--use-srfi' stage of
startup, if that isn't already the case.


> +      (define %default-import-size
> +        ;; This should be the size of the pre-module obarray.
> +        500)

Which also ends up being a minimum size of course ...

> +/* Copy the given alist, i.e., duplicate all its pairs recursively.  */
> +static inline SCM
> +alist_copy (SCM alist)

If you end up using this you could move scm_srfi1_alist_copy from
srfi/srfi-1.c into the core (leaving behind a #define and a
re-export).

> -/*
> -  TODO: should export this function? --hwn.
> - */
> -static SCM
> -scm_export (SCM module, SCM namelist)
> +SCM
> +scm_module_export (SCM module, SCM namelist)

That could be an "scm_i_" (for now at least) if it's not documented.