Re: Future of g-wrap (and guile wrappers in general).

[Matthias Koeppe]

Rob,

thanks for your extensive comparison between g-wrap and SWIG.  I've
always wanted to look into g-wrap in detail but never had the time, so
you've actually saved me a lot of work.

I am going to comment on some of the issues.

>   - In some ways swig and g-wrap make similar decisions wrt how to
>     handle non-native types (types that don't have a native guile
>     representation - i.e. pointers).  Both use smobs, but where swig
>     uses one smob type, g-wrap uses two, and if I understand things
>     correctly, g-wrap treats the types themselves more as first-class
>     objects than swig.
> 
>     g-wrap uses two smobs types, one for all the smobs that actually
>     carry the C-side pointers -- the actual pointer argument/result
>     instances, and one for representing the wrapped types themselves.
>     Each instance of a wrapped pointer on the guile side is a smob
>     which internally has the C pointer and a pointer to the smob that
>     describes wrapped-type.  The type description smob knows the
>     string name of the type and some optional type specific
>     print/equal/GC functions.  These functions have very similar roles
>     to the typical per-smob-type functions.  (So far, we've rarely
>     changed these functions from their defaults, but they could be
>     handy.)

That's a very nice idea that I would like to integrate into SWIG.

The current situation is the following: From the Scheme point of view,
SWIG C pointer types are not first-class objects; they can only be
accessed from C as structs.  However, no extra hash table lookup is
necessary, as the type struct is accessible as a C global: For
instance, the type struct for an "int*" is in the global C variable
`SWIGTYPE_p_int'.  

SWIG currently does not support custom print/equal/mark/free functions
because I never needed them.  However, it should be easy to add them.
(It's in my TODO list for SWIG.)

>     [[You mention that you mainly use the first-class types for
>     pointer coercion.]]
>
>     However I'm open to suggestions for better approaches to this
>     problem.  In the long run, maybe having g-wrap know about the
>     C-side type heirarchy would be safer, but that's a lot more work,
>     and still doesn't make things completely safe.  Further, I've
>     tended to think of g-wrap as a low-level tool that's sometimes
>     enough by itself, but sometimes requires a bit of guile-side code
>     to wrap its results up completely elegantly.  Overall this issue
>     is still up in the air.
> 
>     In any case, it wasn't immediately obvious that swig could handle
>     the kind of coercions described above.  Can it do something
>     similar?

SWIG keeps track of the C-side type hierarchy.  When a pointer value
is passed to a function, SWIG checks whether it is compatible and
coerces it automatically.  For C++ pointers to derived classes, it
invokes the casting function onto the base class, as this may be
required if multiple inheritance is used.

There is no automatic support for "forced coercions".  If you wanted
to cast an "int*" to a "double*", you would have to make an
appropriate identity function and wrap it:

        %inline %{
           double *foo(int *p) { return (double*) p; }
        %}

>   - In g-wrap, you can define new types if the current set aren't
>     sufficient.  In fact, all of the g-wrap "built in" wrappable types
>     are defined using the same g-wrap functions a user would use to
>     wrap their own types.  While the syntax is a bit wordy, it's
>     extremely flexible, and using the predefined types as an example,
>     you can easily define plain pointer types, or fancier types that
>     handle a nearly arbitrarily complex conversion from the C type to
>     the SCM type and back.  So you can fairly easily define
>     <my:foo-scmlist-as-GList-of-Bar*>, etc.
> 
>     Though it seems like swig allows something similar, g-wrap fully
>     expects that end-users, not just backend designers will be
>     defining their own custom types.  Does swig encourage that as
>     well, or are users who go crazy with sophisticated typemaps in
>     danger of getting themselves in trouble?

It used to be like this in SWIG:  Built-in types were dealt with
explicitly (with a large switch() statement in SWIG's C code), and
typemaps (for dealing with user types) were an advanced feature.

However, this seemed very inflexible and unnatural to me, so I dropped
this distinction.  In current SWIG, every C type is handled with
typemaps, and users are encouraged to use typemaps for their work.
The typemaps for the "built-in" types are read from the file
"typemaps.i".  Because typemap definitions tend to be a little
verbose, I am making use of SWIG's built-in extended C preprocessor
there.

>   - All of the default predefined g-wrap integer types are careful to
>     range check their types before conversion.  From the general docs,
>     this doesn't seem to be the case for swig.  Is that right?  How
>     hard would it be for a user to add their own range-checked types?
>     We have cases where either might be appropriate, but we'd rather
>     have checked types as the default.  I.e. we'd like to be able to
>     choose between <gw:int-u4> and <gw:int-u4-checked> or similar when
>     wrapping a function.

Well, I am using the gh_ functions for converting integer types, and I
am currently not very careful about the ranges.  This needs to be
improved.  Since the gh_ API is deprecated, I guess I need to replace
this by calls to some scm_ functions, doing my own range checking.

It is very easy to add user-defined range-checked types, using
typemaps.

>   - g-wrap sets up a NULL <-> #f isomorphism for all if its pointer
>     types.  So NULL on the C side turns into NULL on the C side and
>     vice versa.  This has been quite convenient, and would be a hard
>     assumption to excise from the existing g-wrapped apps.

The empty list takes this role in SWIG (it's so convenient to type
"(null? p)" to check for a NULL pointer ;-), but I could easily change
SWIG to take #f as well.  The behavior can also be changed by the
user, simply by overriding the typemaps for pointer types. 

Here is the typemap that handles all non-void pointer types:

  %typemap(in) SWIGPOINTER * {
    if (SWIG_Guile_GetPtr($source, (void **) &$target, $descriptor))
      scm_wrong_type_arg(FUNC_NAME, $argnum, $source);
  }

To handle with the #f issue:

  %typemap(in) SWIGPOINTER * {
    if (SCM_FALSEP($source)) $target = NULL;
    else if (SWIG_Guile_GetPtr($source, (void **) &$target, $descriptor))
      scm_wrong_type_arg(FUNC_NAME, $argnum, $source);
  }

>   - From the swig general documentation, I was a little unclear about
>     how floating point numbers are handled.  I'd like for the person
>     writing the wrappers to be able to be very specific about which
>     type(s) they get, and I wasn't sure that was the case in swig.

I am just using gh_scm2double, gh_double2scm to convert between C and
Scheme floats.

>   - g-wrap has a type called <gw:symbol-as-string> which lets you wrap
>     a C function that returns or accepts a string such that these
>     values will automatically be converted to/from symbols on the
>     guile side.  Not required, but convenient.

That's a trivial exercise with typemaps.

>   - using g-wrap's type definition mechanism, we were able to define a
>     number of string types that make sure that we can properly
>     integrate with mixed api's that use both gmalloc and malloc for
>     example.  <gw:m-chars-caller-owned>, <gw:m-chars-callee-owned>,
>     <gw:g-chars-caller-owned>, and <gw:g-chars-callee-owned>, for
>     malloced and gmalloced, caller/callee owned strings respectively.
>     This lets you match a C API very closely, specifying exactly
>     what's required of the wrappers.

I would simply make a typedef for those types:

        typedef char *chars_caller_owned;
        typedef char *chars_callee_owned;  etc

use these type names in the API and give the appropriate typemaps to
those types.  No problem here.

>   - g-wrap maps C chars straight to scm chars.  Swig's handling (as
>     single character strings on the guile side) seems a little less
>     convenient.

That was a bug in 1.3.6 (and earlier); it is fixed for the upcoming
version 1.3.7 (to appear this week).

>   - I was a little confused about how swig handles enumerations.  I've
>     posted before about how g-wrap handles them (always return ints,
>     accept int or symbol, and define enum-specific converters).  In
>     particular, it seemed like swig might the values of the enums to
>     be known at swig time rather than compile time.  Any idea if
>     that's true?

Enums are just integers for SWIG, and Enum declarations just create
integer constants.  There is no support for passing symbols where
enums (i.e., integers) are expected.  

The user can add this support for individual enums by writing
typemaps.  I would like to add some automatic support, but I haven't
made up my mind yet.

That SWIG uses the "values" of enums and other constants in the
wrapper, rather than referring to the constants themselves, is a
quirky feature indeed.  I actually don't know the reason for this.

>   - I noticed in the changelog that swig wasn't using scm_bits_t
>     because of an older guile compatibility issue.  Shouldn't is use
>     scm_bits_t unless it detects an older guile?

I have tried to avoid emitting wrapper code that depends on the Guile
wrappers, because that would make for a maintenance nightmare.  SWIG's
output works with Guile 1.3.4 through Guile 1.5.x.  I guess I am
forced to make an incompatible change when the gh_ interface is no
longer available, but not earlier.

>   - How good is swig's preprocessor.  Unless it's acutally the *same*
>     one you're using to compile your code, relying on it seems a
>     little dangerous to me - possibly giving a false sense of
>     security.
> 
>   - and if you don't rely on the preprocessor, it seems like, given
>     swig's comments about %typedefs, that you'd have to maintain
>     parallel, manual defs for things like size_t.  In g-wrap, since
>     it's spec files are independent of the C code, you have to do that
>     anyhow, but there quite a few predefined types for the common
>     cases.

SWIG has its own (extended) C preprocessor, it is not using the C
compiler's preprocessing step.  I am using it extensively, and I think
it is very stable.  

I would always use the preprocessor, so that I can use the same header
file both for the C compiler and SWIG.  I think this is where SWIG's
power comes from: You just need to maintain one (annotated) interface,
which defines the API both for C and higher-level languages.

>   - In swig variable names affect behavior.  i.e. it seems to
>     encourage defining "typemaps" that include the variable name so
>     that, for example, if you have a typemap named "foo_type *bar",
>     any function that has an arg of type foo_type*, named bar will get
>     special treatment.  This *really* bothers me.  What if I'm trying
>     to wrap an API that just so happens to name its variables of a
>     given type using some of the "special" names, but since I haven't
>     seen all the included typemaps, I have no idea that that's going
>     to invoke special behaviors?

All the standard typemaps are read from "typemaps.i", so you can see
what is defined by reading this file.  The magic arg names are OUTPUT,
INPUT, BOTH and INOUT.

>     I think I'd rather see these special behaviors invoked by
>     annotations for the function (perhaps in comments).  i.e. either
>     in the function's wrapper definition, or if you're wrapping
>     headers you can't modify, as a separate statement in the swig spec
>     file like "%makeoutputarg foo_func some_arg_name" or similar.  I
>     couldn't tell for sure if swig supported that, but if so, it would
>     be nice to be able to turn off the arg name based mapping for a
>     given swig module, for those of us who are paranoid :>

The arg-name-based mapping cannot be turned off.

However, I have plans to move from this variable-name annotation
(which is annoying for other reasons as well) to a real type
annotation.

>   - swig handles multiple return values quite well it looks like,
>     though it might be nice to be able to specify the handling
>     per-function when desired.  OTOH g-wrap doesn't handle multiple
>     return values at all yet, and can't automatically handle "OUTPUT"
>     pointer args like swig can.  (Actually, I suppose swig might
>     already allow you to switch multiple-return methods within a file,
>     dunno...)

The method can be switched at any point of a file.  SWIG 1.3.7 will
provide new sugar macros for this:

        %multiple_values, %values_as_list, %values_as_vector

In 1.3.6, it's a little more verbose (see internals.html).

>   - g-wrap tries to support the idea of shared independent modules.
>     i.e. at least originally, it was intended that someone should be
>     able to use g-wrap to enable (use-modules (opengl)) or
>     (use-modules (glib)) and have other, completely independently
>     created g-wrapped modules use opengl and glib and pass wrapped
>     pointers like <glib:Glist*> around among themselves safely.  Does
>     swig also handle that kind of thing?  It looked like it might,
>     given libswigguile.

Yes, and the partial C type hierarchies defined in the individual
modules are merged properly.

 * * * 

Some more comments.

1) I don't like the idea of having a wrapper generator create "low
   level" procedures and adding an extra level of glue on the Scheme
   side.  In the project where I use SWIG, I always design a C API
   that maps to Scheme procedures with a convenient interface; I try
   to deal with all conversion issues by writing appropriate typemaps.

2) There are dozens of mutually incompatible foreign-function
   interfaces for the various Scheme and Lisp implementations and
   other languages (see Reini Urban's "Design Issues for Foreign
   Function Interfaces").  Using SWIG in a hybrid Scheme/C project may
   make moving between Scheme implementations easier to handle.
   (Currently, SWIG also supports MzScheme.)

3) Since the gh_ interface has been deprecated, I was thinking about
   taking it over and turning it into an API for writing
   cross-implementation typemaps for SWIG.  However, I would
   definitely need contributors for such a project.

4) A foreign function interface like g-wrap that is configurable in
   the native language is a nice thing to have.  However, I think it
   would be most useful if one could import foreign functions
   dynamically in the run time of the Scheme system.  g-wrap, on the
   other hand, creates C wrapper code which need to be compiled and
   linked in.

Cheers,

-- 
Matthias Köppe -- http://www.math.uni-magdeburg.de/~mkoeppe
SWIG makes Guile wrappers for C/C++ libs -- http://www.swig.org
ILISP does module-aware Emacs/Guile interaction -- http://sf.net/projects/ilisp