Re: [Chicken-users] [Swig-dev] Future of SWIG chicken module

[John Lenz]

On 2004.01.28 12:55, felix wrote:
> On Mon, 26 Jan 2004 16:06:15 -0600, John Lenz <address@hidden 
> edu> wrote:
>
> Hi, John!

Hey!

First off, Dave temporarly disabled the SWIG mailing list while that  
email virus was going around because I guess the mailing list was  
getting hit pretty hard.  It's back online now.

> So I propose keeping the built-in parser for the quick-and-dirty  
> hacks
> and to further maintain the SWIG module.
>
> The idea of passing the FFI code to SWIG is intriguing. A simple
> command-line option would do the job.
>
> One more question: does SWIG support callbacks?

Yes and no.

Actually, it depends on what you mean by callbacks :)

SWIG almost exclusivly deals with the scripting language-->C/C++  
direction... that is allowing scripting languages to make calls to C or  
C++ functions.  SWIG takes the viewpoint that usually the C or C++ code  
being wrapped is in some sense "lower level", and if you are writing C  
or C++ code that needs to make calls into say python, you just need to  
use the interfaces and APIs provided by python.  Guile provides a bunch  
of library functions that allow you to make calls from C into scheme,  
and SWIG doesn't even know or care about them.  That is, SWIG does not  
wrap scripting language functions so that you can just call them from  
C.

Secondly, if you mean passing around function pointers and such, SWIG  
somewhat suppots that.  See the following page for a description.
http://www.swig.org/Doc1.3/SWIG.html#n30

Thirdly, if by callbacks you mean virtual functions, then yes. A recent  
addition to SWIG is director support or cross language polymorphism.   
Here, when wrapping a C++ class that has virtual methods, if a class is  
derived from it in say python and some of the virtual functions are  
implemented in python, then when calling the virtual function from C++  
the actual python function will be called.  This is kinda like  
callbacks.  Now the support is in SWIG for this, but the chicken module  
has not implemented it yet.  (This is one of the things I will be  
looking at implementing).  See
http://www.swig.org/Doc1.3/Java.html#n40
http://www.swig.org/Doc1.3/Python.html#n32

Lastly, the reason swig does not allow just any scripting language  
function to be passed to a C/C++ function that takes a function pointer  
is that a callback wrapper function would need to be exported.  But  
since chicken already supports generating this wrapper function with  
(define-callback-wrapper ...) it would be easy to make the swig chicken  
module completly support callbacks.  Again, swig itself does not  
support exporting the callback wrapper function but we wouldn't need it  
to :)


> > And then if it is worth maintaining the SWIG chicken module, there  
> > are  a bunch of technical decisions to be made on exactly how SWIG  
> > and  chicken will interact, which I will discuss in more detail if  
> > we choose  this option.
>
> Well, I would be delighted to help. I don't understand the SWIG code,
> though (from the little I've read so far), and the SWIG internals  
> look pretty complicated to me. But for questions regarding Chicken's  
> side of the fence, I'll be happy to assist.


The big technical decision to be made is "Which program will export the  
wrapper functions/How much wrapping will be done?"

Every scripting language module in swig (including how the current  
chicken module is implemented) generates all the wrapping functions.   
This is neccissary for a lot of languages because to register a  
function with the interpreter it must be in the form (for python)
PyObject *_wrapWhatever(PyObject *self, PyObject *args) {...}
SWIG then sets up this function to extract the C variables from the  
args and self pointers and wrap up the return value into a PyObject *

The current swig chicken module does the same thing... a function like
void _wrapWhatever(int argc, C_word closure, C_word continuation,  
C_word arg1) {...}
and then the C interface is used (like C_fix C_unfix and such).  In the  
current swig chicken module, chicken is viewed exactly as another  
scripting language.

So in any case, I notice that a lot of these wrapper functions can be  
generated by chicken with say the (define-foreign-lambda ) or whatnot.   
So my idea was to not have SWIG generate any wrapper functions at  
all...  After thinking about it this past week I am having doubts as to  
if this will work, but here is the idea anyway.

Swig would take as input the header files and such and would generate a . 
scm file that contained instructions to chicken about the interface.   
This way we can completly avoid generating any wrapper functions at  
all.

You see, the problem really comes down to 1) How types will be  
converted and 2) how linkage will occur.  Since chicken itself is  
exporting to C code, we have a lot more flexibility than a normal  
scripting language.  By giving chicken a description of the interface,  
the chicken generated code can make calls directly into the functions  
being wrapped.

The only problem is support for more complicated types, like say std:: 
vector<int>.  SWIG has the ability to automaticly convert a function  
that say returns a std::vector<int> into a scheme list (the guile swig  
module already does).  This requires generating some wrapper  
function... or at least some wrapper code after the call returns to  
create the scheme list.  SWIG has support for what are called typemaps,  
which allow arbitrary code to be inserted into the wrapper function to  
transform the types.  So SWIG has a typemap that converts a std:: 
vector<int> to a guile scheme list.

Another problem involved with linkage is the runtime type handeling.  
Usually, a struct or a class is stored in say the PyObject * or a  
C_word as a void pointer and some type information.  Thus, in the  
generated _wrap function we need to check is the input argument a valid  
type (since we will eventually cast the pointer from a void * to  
whatever type the function takes).  This is currently where both the  
SWIG chicken module and the (foreign*) functions in chicken are BOTH  
broken.  There is no type checking going on at all.  I can pass a  
pointer of any type to any function which accepts a pointer.  SWIG has  
a bunch of code to do run time type checking... is the input type the  
same as the output type, or can the input type be converted to the  
output type, etc. which would also be lost.  Lastly, even if the user  
is careful and only passes correct types to correct functions, type  
conversion is neccissary with multiple inheritance since the offset of  
the base class inside the derived class might not always be at the  
beginning, so a direct cast from a void * causes problems.

For both of these two things we could obviously add some support for in  
chicken, but we would in a sense be wasting all the time and  
duplicating the support already in SWIG.  SWIG has an extensive amount  
of code dealing with run time type checking... you can have typedefs of  
derived classes and more typedefs and on and on and SWIG will figure it  
all out.  SWIG has a lot of code for some of the more compilcated C++  
types like std::vector and std::list as well.

So maybe a combination of the two?
Say chicken could have a directive or function or something that would  
allow arbitrary code to be insterted before the actuall foreign call is  
made?  This way, SWIG would parse the interface file and SWIG would  
generate all the type conversion code (like the C_fix call on the  
return value or the C_c_pointer_or_null() call on the input.  Thus,  
SWIG would in essense do all the Chicken<->C interface stuff but  
instead of dumping it all in wrapper functions, it would just tell  
chicken about it... That is, SWIG would give chicken the exact C code  
that needed to go before and after the call to the foreign function.   
This way we still avoid generating wrapper functions... all the code  
would in some sense be "inlined" right into the generated chicken c  
code.
Optionally as well, SWIG would only need to export that code for  
"problem" types like pointers to foreign types, c++ types std::vector,  
etc.  The base case like a function that takes integers and returns a  
float or something could stay exactly how it is.

Of course, the other option is to keep the current method of type  
conversion and linkage, that is, SWIG generates a bunch of wrapper  
functions and marshals every type from chicken->c and from c->chicken  
without chicken knowing anything about it.

John