Re: Future of g-wrap (and guile wrappers in general).

[Rob Browning]

Matthias Koeppe <address@hidden> writes:

> Looks like something is wrong with the packaging.  I have made the
> relevant part of internals.html available here:
>
>         http://www.math.uni-magdeburg.de/~mkoeppe/guile/swig-guile.html

OK.  Thanks.  Now that I have had a chance to look at that and all of
the other swig docs, I had some questions and comparisons that I think
will help me (and perhaps everyone else) decide if g-wrap should
continue to exist.

My main question is not "what's the better tool", but rather, "Is
g-wrap solving a sufficiently different problem from swig with a
sufficiently different audience that it should continue to exist and
focus on adressing that audience's needs better, or would swig be an
adequate substitute?"

In truth, there is at least some incentive to drop g-wrap and go with
swig.  That way there would be more development effort to go around,
and enhancements to swig (and the guile backend) would benefit other
projects as well.  It would also eliminate an admittedly obscure
dependency for gnucash.

However, at this point I'm kind of leaning toward the conclusion that
at least for now, (a) it's probably less effort to update g-wrap and
continue using it than to switch, and (b), g-wrap may still be of
benefit, providing features that are useful and might not be
appropriate for a more general wrapper tool like swig.

That said, if you've got the time and energy for the following
summary, I'd like to see what you (and everyone else) think(s) about
the following comments, questions, and observations.  Feel free to
ignore any bits you don't have time for, don't know about, etc.  Also
below I describe g-wrap's current state, but I'm very open to, and
interested in alternatives/suggestions/whatever

But first, no matter what I decide.  I'd like to say thanks for all
the swig work.  swig's guile support is *far* better than it was last
time I looked, and it was kind of interesting to see that g-wrap and
swig's guile backend have made some very similar decisions.

Anyway, the following questions and comparisons aren't sorted in any
particular way, but they cover the salient "swig vs. g-wrap issues"
that I've thought of.

  - It looks like swig's handling of the module system is quite good,
    and more up to date than g-wrap's ATM.

  - In some ways swig and g-wrap make similar decisions wrt how to
    handle non-native types (types that don't have a native guile
    representation - i.e. pointers).  Both use smobs, but where swig
    uses one smob type, g-wrap uses two, and if I understand things
    correctly, g-wrap treats the types themselves more as first-class
    objects than swig.

    g-wrap uses two smobs types, one for all the smobs that actually
    carry the C-side pointers -- the actual pointer argument/result
    instances, and one for representing the wrapped types themselves.
    Each instance of a wrapped pointer on the guile side is a smob
    which internally has the C pointer and a pointer to the smob that
    describes wrapped-type.  The type description smob knows the
    string name of the type and some optional type specific
    print/equal/GC functions.  These functions have very similar roles
    to the typical per-smob-type functions.  (So far, we've rarely
    changed these functions from their defaults, but they could be
    handy.)

    For example, if you define a wrapped "pointer" type in g-wrap
    named <gw:Foo*>, then at runtime, a new smob will be created to
    represent the type, and it will be bound to an actual guile
    variable named <gw:Foo*> (let's ignore the naming convention for
    the moment -- it may or may not be the best one).  Then whenever
    g-wrap needs to create an instance of a <gw:Foo*>, either because
    it's a return value from or argument to a wrapped function, it'll
    create a smob of the smob type used to represent all g-wrapped
    pointers, and that smob instance will have an internal pointer to
    the smob representing the <gw:Foo*> type.

    You might wonder why this was done.  By having an actual smob
    representing the wrapped type, and having it bound at the guile
    level to an actual variable, it's possible to do things that are
    often necessary like conversions:

      (gw:coerce-wptr some-obj <gw:Bar*>) -> g-wrapped-Bar*

    efficiently, without any extra hash-table lookup, etc.  You might
    also wonder when this type of thing would really be useful.
    Consider one of our helper functions:

      (gw:map-glist some-func glist-of-bar-ptrs <gw:Bar*>)

    this is a function that will allow you to map over an arbitrary
    <glib:GList*>.  Normally you wouldn't be able to do this, since
    guile has no idea what kind of ptrs are being stored inside the
    glist, but in cases where you know the glist is homogeneous, you
    can use this function to tell gw:map-glist what kind of smobs to
    create and give to some-func.  Though potentially risky (as are
    any C-side type coercions), this has been extremely useful.

    However I'm open to suggestions for better approaches to this
    problem.  In the long run, maybe having g-wrap know about the
    C-side type heirarchy would be safer, but that's a lot more work,
    and still doesn't make things completely safe.  Further, I've
    tended to think of g-wrap as a low-level tool that's sometimes
    enough by itself, but sometimes requires a bit of guile-side code
    to wrap its results up completely elegantly.  Overall this issue
    is still up in the air.

    In any case, it wasn't immediately obvious that swig could handle
    the kind of coercions described above.  Can it do something
    similar?

  - In g-wrap, you can define new types if the current set aren't
    sufficient.  In fact, all of the g-wrap "built in" wrappable types
    are defined using the same g-wrap functions a user would use to
    wrap their own types.  While the syntax is a bit wordy, it's
    extremely flexible, and using the predefined types as an example,
    you can easily define plain pointer types, or fancier types that
    handle a nearly arbitrarily complex conversion from the C type to
    the SCM type and back.  So you can fairly easily define
    <my:foo-scmlist-as-GList-of-Bar*>, etc.

    Though it seems like swig allows something similar, g-wrap fully
    expects that end-users, not just backend designers will be
    defining their own custom types.  Does swig encourage that as
    well, or are users who go crazy with sophisticated typemaps in
    danger of getting themselves in trouble?

  - The disadvantage to the above is that with g-wrap you *do* have to
    define everything.  You have to define a g-wrap wrapped type for
    every pointer type you want <my:Account*>, <my:Foo*>, <my:Bar*>,
    etc.  whereas I think swig handles these pointer types implicitly.
    These defs are usually trivial copy-pastes, but perhaps a
    nuisance.  This could be automated more fairly easily, but right
    now you have to be somewhat painfully explicit.

  - All of the default predefined g-wrap integer types are careful to
    range check their types before conversion.  From the general docs,
    this doesn't seem to be the case for swig.  Is that right?  How
    hard would it be for a user to add their own range-checked types?
    We have cases where either might be appropriate, but we'd rather
    have checked types as the default.  I.e. we'd like to be able to
    choose between <gw:int-u4> and <gw:int-u4-checked> or similar when
    wrapping a function.

  - g-wrap sets up a NULL <-> #f isomorphism for all if its pointer
    types.  So NULL on the C side turns into NULL on the C side and
    vice versa.  This has been quite convenient, and would be a hard
    assumption to excise from the existing g-wrapped apps.

  - From the swig general documentation, I was a little unclear about
    how floating point numbers are handled.  I'd like for the person
    writing the wrappers to be able to be very specific about which
    type(s) they get, and I wasn't sure that was the case in swig.

  - g-wrap has a type called <gw:symbol-as-string> which lets you wrap
    a C function that returns or accepts a string such that these
    values will automatically be converted to/from symbols on the
    guile side.  Not required, but convenient.

  - using g-wrap's type definition mechanism, we were able to define a
    number of string types that make sure that we can properly
    integrate with mixed api's that use both gmalloc and malloc for
    example.  <gw:m-chars-caller-owned>, <gw:m-chars-callee-owned>,
    <gw:g-chars-caller-owned>, and <gw:g-chars-callee-owned>, for
    malloced and gmalloced, caller/callee owned strings respectively.
    This lets you match a C API very closely, specifying exactly
    what's required of the wrappers.

  - g-wrap maps C chars straight to scm chars.  Swig's handling (as
    single character strings on the guile side) seems a little less
    convenient.

  - I was a little confused about how swig handles enumerations.  I've
    posted before about how g-wrap handles them (always return ints,
    accept int or symbol, and define enum-specific converters).  In
    particular, it seemed like swig might the values of the enums to
    be known at swig time rather than compile time.  Any idea if
    that's true?

  - I noticed in the changelog that swig wasn't using scm_bits_t
    because of an older guile compatibility issue.  Shouldn't is use
    scm_bits_t unless it detects an older guile?

  - How good is swig's preprocessor.  Unless it's acutally the *same*
    one you're using to compile your code, relying on it seems a
    little dangerous to me - possibly giving a false sense of
    security.

  - and if you don't rely on the preprocessor, it seems like, given
    swig's comments about %typedefs, that you'd have to maintain
    parallel, manual defs for things like size_t.  In g-wrap, since
    it's spec files are independent of the C code, you have to do that
    anyhow, but there quite a few predefined types for the common
    cases.

  - In swig variable names affect behavior.  i.e. it seems to
    encourage defining "typemaps" that include the variable name so
    that, for example, if you have a typemap named "foo_type *bar",
    any function that has an arg of type foo_type*, named bar will get
    special treatment.  This *really* bothers me.  What if I'm trying
    to wrap an API that just so happens to name its variables of a
    given type using some of the "special" names, but since I haven't
    seen all the included typemaps, I have no idea that that's going
    to invoke special behaviors?

    I think I'd rather see these special behaviors invoked by
    annotations for the function (perhaps in comments).  i.e. either
    in the function's wrapper definition, or if you're wrapping
    headers you can't modify, as a separate statement in the swig spec
    file like "%makeoutputarg foo_func some_arg_name" or similar.  I
    couldn't tell for sure if swig supported that, but if so, it would
    be nice to be able to turn off the arg name based mapping for a
    given swig module, for those of us who are paranoid :>

  - swig handles multiple return values quite well it looks like,
    though it might be nice to be able to specify the handling
    per-function when desired.  OTOH g-wrap doesn't handle multiple
    return values at all yet, and can't automatically handle "OUTPUT"
    pointer args like swig can.  (Actually, I suppose swig might
    already allow you to switch multiple-return methods within a file,
    dunno...)

  - g-wrap tries to support the idea of shared independent modules.
    i.e. at least originally, it was intended that someone should be
    able to use g-wrap to enable (use-modules (opengl)) or
    (use-modules (glib)) and have other, completely independently
    created g-wrapped modules use opengl and glib and pass wrapped
    pointers like <glib:Glist*> around among themselves safely.  Does
    swig also handle that kind of thing?  It looked like it might,
    given libswigguile.

  - swig handles structs and variables whereas g-wrap doesn't at all
    yet, at least not with any kind of automation.  You'd have to
    create C-side accessors yourself and wrap those ATM.

Thanks

-- 
Rob Browning
rlb @defaultvalue.org, @linuxdevel.com, and @debian.org
Previously @cs.utexas.edu
GPG=1C58 8B2C FB5E 3F64 EA5C  64AE 78FE E5FE F0CB A0AD