C
Charlie Savage
Hi everyone,
I've recently gotten up to speed with SWIG as I try to create Ruby
bindings for the GDAL open source project. SWIG Ruby supports seems
pretty good with one exception - memory management. I have run into
three issues:
* There is no support for telling SWIG that ownership of an object has
changed
* There is no built in infrastructure for making it easy to implement a
mark function
* Multiple Ruby objects can wrap the same C++ object / C struct.
I describe these issues in more detail in these two posts:
http://mailman.cs.uchicago.edu/pipermail/swig-dev/2005-August/014991.html
http://mailman.cs.uchicago.edu/pipermail/swig-dev/2005-August/014992.html
Since these seem to be problems that many people run into when writing
Ruby extensions with SWIG, I have spent some time implementing generic
solutions for them and would like to run them by the list for
comment/feedback/suggestions.
API
---------
The basic api is:
/* Add a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyAddMapping(void* ptr, VALUE object);
/* Get the Ruby object that owns the specified C/C++ struct */
static VALUE SWIG_RubyInstanceFor(void* ptr);
/* Remove a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyRemoveMapping(void* ptr);
The SWIG wrapper code takes care of calling RubyAddMapping and
RubyRemoveMapping when Ruby objects in extensions are created/deleted.
Thus the extension programmer generally only has to call
SWIG_RubyInstanceFor when implementing a mark function (although in some
cases you may have to call SWIG_RubyRemoveMapping if your are manually
destroying objects on the C++ side).
Implementation
----------------
To solve issues #2 and #3 requires keeping track of mappings between
C/C++ structs and Ruby objects. Reading through various newsgroups I
have seen two general approaches - have a hash table that stores these
mappings or add an field to each C/C++ struct to store a reference to a
Ruby object.
As can be seen in the code below, I have gone down the hashtable route
with a slight twist - I use a Ruby hash table from C to store these
mappings. I did this instead of using a C++ map since I wanted to the
code to be just C and it was easier then implementing a hash table from
scratch in C.
Performance
---------------
To make sure using a Ruby hash table performed adequately, I ran some
performance tests.
Test - Create 100,000 Animal (implemented in an extension, taken from
the SWIG Ruby manual). The test machine was a Pentium M Thinkpad T41
laptop.
Test Time
No mappings 9.3 to 9.6 seconds
Mappings using Ruby hash 9.5 to 9.9 seconds
Mappings using std::map 30 seconds
You can see there is slight performance hit for tracking mappings
between C++ objects and Ruby object. Note the C++ map was so slow
because deletions took a lot of time (and there were 100,000 deletetions
by the end of the test run).
Summary
---------
If people think it would be a good idea to add this functionality to
SWIG (whether using this implementation or some other one) then I'd be
glad to take the lead on this.
Thanks,
Charlie
Code
---------------------
This is a new file called rubymappings.swg
/* Global Ruby hash table to store mappings from C/C++
structs to Ruby Objects. */
static VALUE swig_ruby_mappings;
/* Get a Ruby number to reference a pointer */
static VALUE SWIG_RubyPtrToReference(void* ptr) {
/* We cast the pointer to an unsigned long
and then store a reference to it using
a Ruby number object. */
/* Convert the pointer to a Ruby number */
unsigned long value = (unsigned long) ptr;
return LONG2NUM(value);
}
/* Get a Ruby number to reference an object */
static VALUE SWIG_RubyObjectToReference(VALUE object) {
/* We cast the object to an unsigned long
and then store a reference to it using
a Ruby number object. */
/* Convert the Object to a Ruby number */
unsigned long value = (unsigned long) object;
return LONG2NUM(value);
}
/* Get a Ruby object from a previously stored reference */
static VALUE SWIG_RubyReferenceToObject(VALUE reference) {
/* The provided Ruby number object is a reference
to the Ruby object we want.*/
/* First convert the Ruby number to a C number */
unsigned long value = NUM2LONG(reference);
return (VALUE) value;
}
/* Add a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyAddMapping(void* ptr, VALUE object) {
/* In a Ruby hash table we store the pointer and
the associated Ruby object. The trick here is
that we cannot store the Ruby object directly - if
we do then it cannot be garbage collected. So
instead we typecast it as a unsigned long and
convert it to a Ruby number object.*/
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
/* Get a reference to the Ruby object as a Ruby number */
VALUE value = SWIG_RubyObjectToReference(object);
rb_hash_aset(swig_ruby_mappings, key, value);
}
/* Get the Ruby object that owns the specfied C/C++ struct */
static VALUE SWIG_RubyInstanceFor(void* ptr) {
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
/* Now lookup the value stored in the Ruby hash table */
VALUE value = rb_hash_aref(swig_ruby_mappings, key);
if (value == Qnil) {
return Qnil;
}
else {
/* Convert this value to Ruby object */
return SWIG_RubyReferenceToObject(value);
}
}
/* Remove a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyRemoveMapping(void* ptr) {
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
VALUE object = SWIG_RubyInstanceFor(ptr);
/* Reset the C/C++ data struct associated with the Object.
This is needed in case a Ruby object exists longer than
its underlying C++ object. By setting the data_struct
to nil, code in SWIG_Ruby_ConvertPtr can detect this problem
and return an error message, as opposed to causing a
segmentation fault.*/
if (object != Qnil) {
DATA_PTR(object) = 0;
}
/* Now delete the object from the hash table. To
do this we need to call the Hash.delete method
in Ruby. */
static VALUE delete_function = rb_intern("delete");
rb_funcall(swig_ruby_mappings, delete_function, 1, key);
}
/* Setup a Ruby hash table to store mappings */
void SWIG_RubyInitializeMappings() {
/* Create a ruby hash table to store mappings from C++
objects to Ruby objects. Also make sure to tell
the garabage collector about the hash table */
swig_ruby_mappings = rb_hash_new();
rb_gc_register_address(&swig_ruby_mappings);
}
Patches to Hook Into SWIG
----------------------
Index: ruby.cxx
===================================================================
RCS file: /cvsroot/swig/SWIG/Source/Modules/ruby.cxx,v
retrieving revision 1.78
diff -u -i -r1.78 ruby.cxx
--- ruby.cxx 17 Aug 2005 20:35:53 -0000 1.78
+++ ruby.cxx 18 Aug 2005 18:41:22 -0000
@@ -465,6 +465,9 @@
NIL);
Printf(f_init,"\n");
+ // Initialize mapping code
+ Printf(f_init,"SWIG_RubyInitializeMappings();\n");
+
Language::top(n);
/* Finish off our init function */
@@ -1098,7 +1107,8 @@
if (current == CONSTRUCTOR_INITIALIZE) {
String *action = Getattr(n,"wrap:action");
if (action) {
- Append(action,"DATA_PTR(self) = result;");
+ Append(action,"DATA_PTR(self) = result;\n");
+ Append(action,"SWIG_RubyAddMapping(result, self);\n");
}
}
emit_action(n,f);
@@ -1850,6 +1860,8 @@
Printf(freebody, "free((char*) %s);\n", Swig_cparm_name(0,0));
}
}
+ Printf(freebody, tab4, NIL);
+ Printf(freebody, "SWIG_RubyRemoveMapping(%s);\n", Swig_cparm_name(0,0));
Printv(freebody, "}\n", NIL);
Printv(f_wrappers, freebody, NIL);
Index: ruby.swg
===================================================================
RCS file: /cvsroot/swig/SWIG/Lib/ruby/ruby.swg,v
retrieving revision 1.29
diff -u -i -r1.29 ruby.swg
--- ruby.swg 1 Feb 2005 00:08:19 -0000 1.29
+++ ruby.swg 18 Aug 2005 18:44:19 -0000
@@ -6,6 +6,7 @@
%runtime "rubyhead.swg"
%runtime "swigrun.swg" // Common C API type-checking code
+%runtime "rubymappings.swg" /* Stores mappings from C/C++ struct
to Ruby objects */
%runtime "rubydef.swg"
%insert(initbeforefunc) "swiginit.swg"
Index: rubydef.swg
===================================================================
RCS file: /cvsroot/swig/SWIG/Lib/ruby/rubydef.swg,v
retrieving revision 1.17
diff -u -i -r1.17 rubydef.swg
--- rubydef.swg 1 Feb 2005 00:08:20 -0000 1.17
+++ rubydef.swg 18 Aug 2005 18:43:53 -0000
@@ -71,6 +71,13 @@
if (!ptr)
return Qnil;
+
+ /* Have we already wrapped this pointer? */
+ obj = SWIG_RubyInstanceFor(ptr);
+
+ if (obj != Qnil) {
+ return obj;
+ }
if (type->clientdata) {
sklass = (swig_class *) type->clientdata;
I've recently gotten up to speed with SWIG as I try to create Ruby
bindings for the GDAL open source project. SWIG Ruby supports seems
pretty good with one exception - memory management. I have run into
three issues:
* There is no support for telling SWIG that ownership of an object has
changed
* There is no built in infrastructure for making it easy to implement a
mark function
* Multiple Ruby objects can wrap the same C++ object / C struct.
I describe these issues in more detail in these two posts:
http://mailman.cs.uchicago.edu/pipermail/swig-dev/2005-August/014991.html
http://mailman.cs.uchicago.edu/pipermail/swig-dev/2005-August/014992.html
Since these seem to be problems that many people run into when writing
Ruby extensions with SWIG, I have spent some time implementing generic
solutions for them and would like to run them by the list for
comment/feedback/suggestions.
API
---------
The basic api is:
/* Add a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyAddMapping(void* ptr, VALUE object);
/* Get the Ruby object that owns the specified C/C++ struct */
static VALUE SWIG_RubyInstanceFor(void* ptr);
/* Remove a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyRemoveMapping(void* ptr);
The SWIG wrapper code takes care of calling RubyAddMapping and
RubyRemoveMapping when Ruby objects in extensions are created/deleted.
Thus the extension programmer generally only has to call
SWIG_RubyInstanceFor when implementing a mark function (although in some
cases you may have to call SWIG_RubyRemoveMapping if your are manually
destroying objects on the C++ side).
Implementation
----------------
To solve issues #2 and #3 requires keeping track of mappings between
C/C++ structs and Ruby objects. Reading through various newsgroups I
have seen two general approaches - have a hash table that stores these
mappings or add an field to each C/C++ struct to store a reference to a
Ruby object.
As can be seen in the code below, I have gone down the hashtable route
with a slight twist - I use a Ruby hash table from C to store these
mappings. I did this instead of using a C++ map since I wanted to the
code to be just C and it was easier then implementing a hash table from
scratch in C.
Performance
---------------
To make sure using a Ruby hash table performed adequately, I ran some
performance tests.
Test - Create 100,000 Animal (implemented in an extension, taken from
the SWIG Ruby manual). The test machine was a Pentium M Thinkpad T41
laptop.
Test Time
No mappings 9.3 to 9.6 seconds
Mappings using Ruby hash 9.5 to 9.9 seconds
Mappings using std::map 30 seconds
You can see there is slight performance hit for tracking mappings
between C++ objects and Ruby object. Note the C++ map was so slow
because deletions took a lot of time (and there were 100,000 deletetions
by the end of the test run).
Summary
---------
If people think it would be a good idea to add this functionality to
SWIG (whether using this implementation or some other one) then I'd be
glad to take the lead on this.
Thanks,
Charlie
Code
---------------------
This is a new file called rubymappings.swg
/* Global Ruby hash table to store mappings from C/C++
structs to Ruby Objects. */
static VALUE swig_ruby_mappings;
/* Get a Ruby number to reference a pointer */
static VALUE SWIG_RubyPtrToReference(void* ptr) {
/* We cast the pointer to an unsigned long
and then store a reference to it using
a Ruby number object. */
/* Convert the pointer to a Ruby number */
unsigned long value = (unsigned long) ptr;
return LONG2NUM(value);
}
/* Get a Ruby number to reference an object */
static VALUE SWIG_RubyObjectToReference(VALUE object) {
/* We cast the object to an unsigned long
and then store a reference to it using
a Ruby number object. */
/* Convert the Object to a Ruby number */
unsigned long value = (unsigned long) object;
return LONG2NUM(value);
}
/* Get a Ruby object from a previously stored reference */
static VALUE SWIG_RubyReferenceToObject(VALUE reference) {
/* The provided Ruby number object is a reference
to the Ruby object we want.*/
/* First convert the Ruby number to a C number */
unsigned long value = NUM2LONG(reference);
return (VALUE) value;
}
/* Add a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyAddMapping(void* ptr, VALUE object) {
/* In a Ruby hash table we store the pointer and
the associated Ruby object. The trick here is
that we cannot store the Ruby object directly - if
we do then it cannot be garbage collected. So
instead we typecast it as a unsigned long and
convert it to a Ruby number object.*/
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
/* Get a reference to the Ruby object as a Ruby number */
VALUE value = SWIG_RubyObjectToReference(object);
rb_hash_aset(swig_ruby_mappings, key, value);
}
/* Get the Ruby object that owns the specfied C/C++ struct */
static VALUE SWIG_RubyInstanceFor(void* ptr) {
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
/* Now lookup the value stored in the Ruby hash table */
VALUE value = rb_hash_aref(swig_ruby_mappings, key);
if (value == Qnil) {
return Qnil;
}
else {
/* Convert this value to Ruby object */
return SWIG_RubyReferenceToObject(value);
}
}
/* Remove a mapping from a C/C++ struct to a Ruby object */
static void SWIG_RubyRemoveMapping(void* ptr) {
/* Get a reference to the pointer as a Ruby number */
VALUE key = SWIG_RubyPtrToReference(ptr);
VALUE object = SWIG_RubyInstanceFor(ptr);
/* Reset the C/C++ data struct associated with the Object.
This is needed in case a Ruby object exists longer than
its underlying C++ object. By setting the data_struct
to nil, code in SWIG_Ruby_ConvertPtr can detect this problem
and return an error message, as opposed to causing a
segmentation fault.*/
if (object != Qnil) {
DATA_PTR(object) = 0;
}
/* Now delete the object from the hash table. To
do this we need to call the Hash.delete method
in Ruby. */
static VALUE delete_function = rb_intern("delete");
rb_funcall(swig_ruby_mappings, delete_function, 1, key);
}
/* Setup a Ruby hash table to store mappings */
void SWIG_RubyInitializeMappings() {
/* Create a ruby hash table to store mappings from C++
objects to Ruby objects. Also make sure to tell
the garabage collector about the hash table */
swig_ruby_mappings = rb_hash_new();
rb_gc_register_address(&swig_ruby_mappings);
}
Patches to Hook Into SWIG
----------------------
Index: ruby.cxx
===================================================================
RCS file: /cvsroot/swig/SWIG/Source/Modules/ruby.cxx,v
retrieving revision 1.78
diff -u -i -r1.78 ruby.cxx
--- ruby.cxx 17 Aug 2005 20:35:53 -0000 1.78
+++ ruby.cxx 18 Aug 2005 18:41:22 -0000
@@ -465,6 +465,9 @@
NIL);
Printf(f_init,"\n");
+ // Initialize mapping code
+ Printf(f_init,"SWIG_RubyInitializeMappings();\n");
+
Language::top(n);
/* Finish off our init function */
@@ -1098,7 +1107,8 @@
if (current == CONSTRUCTOR_INITIALIZE) {
String *action = Getattr(n,"wrap:action");
if (action) {
- Append(action,"DATA_PTR(self) = result;");
+ Append(action,"DATA_PTR(self) = result;\n");
+ Append(action,"SWIG_RubyAddMapping(result, self);\n");
}
}
emit_action(n,f);
@@ -1850,6 +1860,8 @@
Printf(freebody, "free((char*) %s);\n", Swig_cparm_name(0,0));
}
}
+ Printf(freebody, tab4, NIL);
+ Printf(freebody, "SWIG_RubyRemoveMapping(%s);\n", Swig_cparm_name(0,0));
Printv(freebody, "}\n", NIL);
Printv(f_wrappers, freebody, NIL);
Index: ruby.swg
===================================================================
RCS file: /cvsroot/swig/SWIG/Lib/ruby/ruby.swg,v
retrieving revision 1.29
diff -u -i -r1.29 ruby.swg
--- ruby.swg 1 Feb 2005 00:08:19 -0000 1.29
+++ ruby.swg 18 Aug 2005 18:44:19 -0000
@@ -6,6 +6,7 @@
%runtime "rubyhead.swg"
%runtime "swigrun.swg" // Common C API type-checking code
+%runtime "rubymappings.swg" /* Stores mappings from C/C++ struct
to Ruby objects */
%runtime "rubydef.swg"
%insert(initbeforefunc) "swiginit.swg"
Index: rubydef.swg
===================================================================
RCS file: /cvsroot/swig/SWIG/Lib/ruby/rubydef.swg,v
retrieving revision 1.17
diff -u -i -r1.17 rubydef.swg
--- rubydef.swg 1 Feb 2005 00:08:20 -0000 1.17
+++ rubydef.swg 18 Aug 2005 18:43:53 -0000
@@ -71,6 +71,13 @@
if (!ptr)
return Qnil;
+
+ /* Have we already wrapped this pointer? */
+ obj = SWIG_RubyInstanceFor(ptr);
+
+ if (obj != Qnil) {
+ return obj;
+ }
if (type->clientdata) {
sklass = (swig_class *) type->clientdata;