Re: [Chicken-users] Chicken and SWIG

[John Lenz]

On 2004.05.03 00:17, Felix Winkelmann wrote:
> John Lenz wrote:
> > Just to keep everyone updated, today I checked the changes to the   
> > chicken module into the swig cvs.  Swig now uses the new type Felix   
> > added, so c++ multiple inheritance works as well as proper type   
> > checking.
> >
> > Because of this, future versions of SWIG will now require chicken   
> > version 1.40 or above.
>
> Brilliant! I'm looking forward to try it out. I would also be
> delighted, if you had any ideas how to integrate SWIG into
> Chicken. So far my approaches appear clumsy. For example:
>
> #>(swig) ... code <#
>
> and then "csc myfile.scm -swig" would generate temporary files for
> the embedded code tagged (swig) and automatically invoke SWIG and
> compile the generated files.
>
> Any suggestions for something cleaner are welcome.

Well, this email is pretty long, so I don't know if this is cleaner :)

Currently, we run swig, and it produces a .c or a .cxx file that needs  
to be compiled, and it produces a couple of .scm files, which then also  
need to be compiled by chicken.

For example, we produce a file called modname.scm that has
(cond-expand ((or chicken-compile-shared shared)) (else (declare (unit  
modname))))

(declare
   (hide swig-init)
   (foreign-declare "C_extern void modname_swig_init(int,C_word, 
C_word) C_noret;"))
(define swig-init (##core#primitive "modname_swig_init"))
(swig-init)

And in code which we want to use the wrapped library, we have a
(uses modname).

What would be cool is for the chicken compiler to automaticly generate  
this call to modname_swig_init, and include it in the current module.   
That way we would not need to generate the above file in SWIG (chicken  
would pass an argument flag like -nounit or something telling SWIG not  
to generate this file).  Thus we would not need an extra run of chicken  
to compile the above trivial code.

So what would happen is that swig would run and produce a modname_wrap. 
cxx file.  This would include a function called modname_swig_init which  
would need to get called.  modname_swig_init just does some basic setup  
and then registers all the wrapped functions.  The chicken compiler  
would automaticly generate this call in the top level environment, and  
thus the rest of that chicken source file would be able to make calls  
to the wrapped functions.  Secondly, if a user wanted to create a unit,  
they could just declare the unit in the file that includes the #>(swig)  
<# stuff.

The only problem is the modname.  SWIG gets the modname from the "% 
module modname" directive in the input.  Should the user select it or  
should chicken just randomly generate it?  The problem is, the module  
name is by default prepended to all symbols.  functions will be called  
"modname:func1" if the function was called func1.  This can be  
overridden by the "-prefix <name>" argument to swig, or by passing the  
-noprefix option.

SWIG will generate a file called modname_wrap.c or modname_wrap.cxx  
depending on if SWIG was called with the -c++ command line argument.   
This can also be changed by the "-o filename" flag to SWIG.  (Note I  
just thought of this, but you will also need some way to pick between c  
and c++... something like #>(swig-c)...<# and #>(swig-c++) .. #<  
perhaps?)

-----------------------------------------------------------------------
Ok, now a little more complicated :)
SWIG also by default generates a modname-clos.scm file (unless the - 
noclos flag is passed), which includes a bunch of (define-class ..) and  
(define-method ..) and whatnot.  Right now, we would just run chicken  
again on this file.

But what would be cool is for chicken to read this file after running  
SWIG.  So then these definitions would also be included directly into  
the "scope" of the chicken input file.

Thirdly, SWIG generates a modname-generic.scm which includes code
(define -set-a!- (make-generic "set-a!")) ;; class <modname:Foo>
...
SWIG does not generate this file if -nogeneric is passed.
-----------------------------------------------------------------------
Ok, I have an idea now... kinda forget everything I said above :)

Ok, how about this?  When SWIG is called with say a -chicken -usestdio  
flag, SWIG accepts the .i file on standard input, produces the  
modname_wrap.cxx file, and then writes all the .scm files to standard  
output?  chicken wouldn't even need to worry about calling the  
modname_swig_init function, because the (foreign-declare ..) and such  
would be right at the top of the standard output from SWIG.  chicken  
would then compile whatever comes from swig standard out. (which would  
include the call to modname_swig_init)

SWIG would remove the first line involving (declare unit) when  
exporting to stdout, but otherwise we would just write the .scm files  
to stdout.  This way we don't generate any temporary files.  The only  
file SWIG generates will be the c or c++ file.  Chicken should probably  
pass the -o name option, and name the output file the same name as the  
file chicken is generating, with say "_swig_wrap" appended or  
something.

Ok, for options, I think chicken should allow the user to select the  
following
1) choose between c and c++ (this coorisponds to the -c++ flag to SWIG.   
No -c++ option means c).
2) optinally allow the user to select the -prefix or -noprefix
3) Allow the user to select -noclos or -nogeneric.  Possibly invert the  
defaults... i.e have chicken default to -noclos and -nogeneric, and  
optionally turn those on.

So maybe something like this?
#>(swig,c,clos,coolprefix) ... #<
would mean swig in c mode, with only the -nogeneric option, and a - 
prefix coolprefix option.

#>(swig,c++) ... #< would mean pass -c++ -nogeneric -noclos -noprefix

This way, chicken should just add a "%module whatever" to swig stdin.   
Because chicken would always pass the -prefix or -noprefix option, the  
module name will never appear anywhere visable to the user.  It will  
only appear in the _wrap.cxx file.  The modname is prepended to  
generated functions, so that we don't clutter up the global symbol  
table.

Note this way we can allow multiple #>(swig) .. <# blocks per file...  
each one would get a different module name, but as long as the module  
names are different, SWIG will be run twice, and if the prefixes are  
different, the generated symbol names won't clash...

------------------------------------------------------------------------
Lastly, and even more complicated, SWIG allows type dependence between  
modules.  That is, if I say wrap a class Foo in one module (mod1), and  
then derive a class Bar : public Foo in another module (mod2). I need  
to include the "%import mod1.i" directive in mod2. SWIG will then look  
in the mod1.i for types and such, and then swig will know that Bar is a  
derived class of Foo.

What happens at runtime is that swig stores a bunch of information  
about types, which includes stuff like which classes are derived from  
which other classes.  In order for type dependence between modules to  
work, the two modules that get loaded must merge together the two type  
trees.  (since the type info for Foo will be in mod1 and the type info  
for Bar will be in mod2)

By default, every module does all the type stuff by itself.  Thus each  
module keeps the type information seperate and everthing is fine and  
dandy.  But if the "-noruntime" option is passed to SWIG, SWIG does not  
include this runtime type managing code.  Instead, the runtime code is  
expected to be available during linking.  Thus, say module 1 will  
include the runtime code AND make it globaly visible (i.e. the  
functions will not be declared "static") by passing the "-runtime"  
option.  module 2 will be passed the "-noruntime" option, and during  
the linking phase, the module2 type functions will be linked to the  
functions in module 1.  Thus the two type trees will be merged together  
and typing works between modules.  (NOTE: if neither -runtime or - 
noruntime is passed, the type functions are included and declared  
static so that other modules can not link to them).

How does this impact chicken?  Well, it would be nice to support this  
as well from the chicken called swig.  Thus we might want to add  
another option that the user can select.
#>(swig,c,clos,coolprefix1,runtime) ... <#
#>(swig,c,clos,coolprefix2,noruntime) ... <#
and chicken would just pass -runtime or -noruntime to swig.  (If  
neither is given, not pass either).

The problem is, SWIG needs the %import directive, because it needs to  
know about the base classes.  Since the code that normally would be in  
the file %import would read, and that code is from some other chicken  
file, the file we would import wouldn't exist.  To solve this, chicken  
would also have to have a similar %import directive.  All this would do  
would be to parse the chicken file looking for the #<(swig...) <#  
directive, (add the SAME %module <modname> that would be added when  
parsing that file normally), and pass that code to swig somehow.  That  
code would either need to be written to a temporary file, or included  
in the swig input file in say a
%import %{
code here
%}
NOTE: we can't just include the code directly in the file, because SWIG  
needs to know if it should be generating wrapper functions for it or  
not.  When we import a file, we don't generate any code, just use it  
for type purposes.  Thus SWIG knows that code in the %import %{ it  
should just be reading for type purposes.

#>(swig,c,coolmod,noruntime,import=mod1.scm,import=mod3.scm)... #<

Note as well, if we allow multiple #>(swig,...) ... <# in the same  
chicken file, and the user wants to have type dependence between them,  
chicken would need some way to do that inclusion as well...  This  
implies that we would need to optionally name each swig block, so that  
the second swig block can reference the first one.
somethine like
#>(swig,...,importblock=otherBlock) <#.
In this case, chicken would do the same thing.. it would include the  
code from then otherBlock in an %import %{ ... %} when calling SWIG for  
this block.  I was thinking we could use the prefix for this naming of  
the blocks, but we run into trouble if no prefix is given i.e. - 
noprefix.  Also would need a way to determine which #>(swig) <# block  
in the included file. Although, it seems kinda wierd to have multiple  
#>(swig..) blocks in the same file... probably just restrict it to one  
per file!

One last problem with this :)
In the mod1-clos.scm file, it will include the (define-class <mod1:Foo> ...)
depending on the status of -prefix and -noprefix.  i.e. if -noprefix is  
passed, it would export a (define-class <Foo> ...).
The problem is in mod2-clos.scm.  mod2-clos.scm will include something  
like
(define-class <coolmod2:Bar> (<mod1:Foo>) ())
Note the use of mod1:  If say when compiling mod1 we passed a -prefix  
coolmod1 option, then mod1-clos.scm will have a definiton <coolmod1: 
Foo> instead of a <mod1:Foo>.  Currently it is a limitation of SWIG  
that there is no way to change this prefix in imported files!!! i.e.  
currently swig will always append the module name of the imported file  
(from the %module directive in the imported file) to the class symbols.   
Note the -prefix option replaces the mod2: with something else, so that  
won't work.

What I think would need to happen is that swig would accept a
%chickenprefix "prefix" directive.  This would have the same effect as  
the -prefix command line argument.  Thus when chicken is generating the  
code to be passed to swig, when it is parsing an import=mod1.scm thing,  
it would add the directive %chickenprefix coolmod1 to the code inside
%import %{
%chickenprefix "coolmod1"
code...
%}
SWIG would then know enough to use the variable <coolmod1:Foo>.
This way again, the actual module name is never seen by the user and  
chicken can pick whatever it wants.

John