[ANN] FFI 0.1.1 (Foreign Function Interface) for Ruby 1.8.6/7 and1.9

[Charles Oliver Nutter]

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:


require 'ffi'

module GetPid
extend FFI::Library

attach_function :getpid, [], :uint
end

puts GetPid.getpid


Here's another, calling qsort and passing a Ruby block as a C callback:


require 'ffi'

module LibC
extend FFI::Library
callback :qsort_cmp, [ ointer, ointer ], :int
attach_function :qsort, [ ointer, :int, :int, :qsort_cmp ], :int
end

p = MemoryPointer.newint, 2)
p.put_array_of_int32(0, [ 2, 1 ])
puts "Before qsort #{p.get_array_of_int32(0, 2).join(', ')}"
LibC.qsort(p, 2, 4) do |p1, p2|
i1 = p1.get_int32(0)
i2 = p2.get_int32(0)
i1 < i2 ? -1 : i1 > i2 ? 1 : 0
end
puts "After qsort #{p.get_array_of_int32(0, 2).join(', ')}"


I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.
Docs are a little slim at this point, so feel free to experiment and
update the JRuby wiki page:

http://wiki.jruby.org/wiki/Calling_C_from_JRuby

I'm sure docs from here will filter back into the library and out into
the general cosmos.

Finally, there's no need to write a C extension to call C libraries, and
the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby 1.1.4+, and
Rubinius (though Rubinius has no callback support yet).

Don't be an extension stooge! Use FFI!

- Charlie

 

[Thomas Hurst]

The JRuby team is proud to announce the release of FFI for Ruby
1.8.6/7 and 1.9!

Finally, there's no need to write a C extension to call C libraries,
and the same FFI code will work in Ruby 1.8.6/7, Ruby 1.9, JRuby
1.1.4+, and Rubinius (though Rubinius has no callback support yet).

Awesome. I've used DL to link up some custom libs to a Ruby service,
will give FFI a go and see how it compares

Are things like structs likely to be supported in future, ala Python
ctypes?

 

[Charles Oliver Nutter]

Awesome. I've used DL to link up some custom libs to a Ruby service,
will give FFI a go and see how it compares

Are things like structs likely to be supported in future, ala Python
ctypes?

Actually structs are already supported! See the blog post, and I believe
there's some examples shipped with the gem. There needs to be more docs,
certainly, and hopefully they'll get some TLC soon.

Also, I forgot to call out Evan Phoenix for coming up with the API and
initial design, and he or someone else on Rubinus wrote up the
templating/header-file-processing stuff as well. And of course a huge
thanks to Wayne Meissner for implementing FFI not just once (for JRuby)
but twice (for C Ruby). His work will mean a huge leap forward in
cross-impl portability.

- Charlie

 

[Rados³aw Bu³at]

T24gU2F0LCBOb3YgMSwgMjAwOCBhdCA2OjQwIEFNLCBDaGFybGVzIE9saXZlciBOdXR0ZXIKPGNo
YXJsZXMubnV0dGVyQHN1bi5jb20+IHdyb3RlOgo+IEFjdHVhbGx5IHN0cnVjdHMgYXJlIGFscmVh
ZHkgc3VwcG9ydGVkISBTZWUgdGhlIGJsb2cgcG9zdCwgYW5kIEkgYmVsaWV2ZQo+IHRoZXJlJ3Mg
c29tZSBleGFtcGxlcyBzaGlwcGVkIHdpdGggdGhlIGdlbS4KCllvdXIgYmxvZyBpcyBwcmVhdHkg
a25vd24gYnV0IGZvciBjbGFyaXR5OgpodHRwOi8vYmxvZy5oZWFkaXVzLmNvbS8yMDA4LzEwL2Zm
aS1mb3ItcnVieS1ub3ctYXZhaWxhYmxlLmh0bWwKOi0pCgotLSAKUmFkb3OzYXcgQnWzYXQKCmh0
dHA6Ly9yYWRhcmVrLmpvZ2dlci5wbCAtIG3zaiBibG9nCg==

 

[Charles Oliver Nutter]

Your blog is preaty known but for clarity:
http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html

Oh right, a link would have been useful. Thank you

- Charlie

 

[Charles Oliver Nutter]

I posted a blog entry with a longer description of the library,
additional examples, and links to some other documentation and posts.

And then I completely forgot to include the blog post URL...

http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html

- Charlie

 

[Luc Heinrich]

And then I completely forgot to include the blog post URL...

Looks cool, great work.

Two questions:

- Are variadic functions supported?
- Do you have any idea or measurements of the overhead of calling
through FFI as opposed to using a compiled extension?

 

[Charles Oliver Nutter]

Looks cool, great work.

Two questions:

- Are variadic functions supported?
- Do you have any idea or measurements of the overhead of calling
through FFI as opposed to using a compiled extension?

Wayne answers the latter, sorta, on his followup blog post:

http://blog.headius.com/2008/10/ffi-for-ruby-now-available.html

He doesn't have specific numbers for performance at the moment, but the
short story is that FFI introduces a bit of overhead; ultimately I
believe that the overhead gets lost in the flow of a Ruby application,
especially when you're tossing units of work across like SQL queries or
arrays. Wayne probably can fill in more details on what the actual
overhead is like.

And I'd also expect that any small amount of overhead is vastly
outweighed by the ability to write an FFI-based library once and use it
across implementations.

- Charlie

 

[Luc Heinrich]

He doesn't have specific numbers for performance at the moment, but
the short story is that FFI introduces a bit of overhead; ultimately
I believe that the overhead gets lost in the flow of a Ruby
application, especially when you're tossing units of work across
like SQL queries or arrays.

# --- [begin unscientific test] --------

require 'rubygems'
require 'benchmark'
require 'zlib'
require 'ffi'
require 'dl/import'

module Zlib_ffi
extend FFI::Library
attach_function :zlib_version, :zlibVersion, [], :string
end

module Zlib_dl
extend DL::Importable
dlload "libz.dylib"
extern "const char* zlibVersion()"
end

puts Zlib.zlib_version
puts Zlib_ffi.zlib_version
puts Zlib_dl.zlibVersion

Benchmark.bm(3) do |bm|
bm.report("ext") { 500_000.times { Zlib.zlib_version } }
bm.report("ffi") { 500_000.times { Zlib_ffi.zlib_version } }
bm.report("dl") { 500_000.times { Zlib_dl.zlibVersion } }
end

# --- [end unscientific test] --------

This gives the following results:

1.2.3
1.2.3
1.2.3
user system total real
ext 1.050000 0.320000 1.370000 ( 1.373800)
ffi 2.160000 0.660000 2.820000 ( 2.818966)
dl 3.500000 1.060000 4.560000 ( 4.552789)

All this using MacPorts MRI 1.8.7-p72 under OS X 10.5.5. The observed
overhead is slightly over 2x for ffi, probably not a big deal unless
ffi calls are used in tight loops I guess.


PS: haven't seen any trace of variadic function support in the code.

 

[Rados³aw Bu³at]

V2hlcmUgY2FuIEkgZmlsZSBhbiBpc3N1ZT8gSSBoYXZlIHRyb3VibGUgdG8gYnVpbGQgaW4gd2l0
aCBydWJ5MS45IG9uIDY0Yml0LgoKLS0gClJhZG9zs2F3IEJ1s2F0CgpodHRwOi8vcmFkYXJlay5q
b2dnZXIucGwgLSBt82ogYmxvZwo=

 

[Rados³aw Bu³at]

PiBUaGlzIGdpdmVzIHRoZSBmb2xsb3dpbmcgcmVzdWx0czoKPgo+IDEuMi4zCj4gMS4yLjMKPiAx
LjIuMwo+ICAgICAgICAgdXNlciAgICAgc3lzdGVtICAgICAgdG90YWwgICAgICAgIHJlYWwKPiBl
eHQgIDEuMDUwMDAwICAgMC4zMjAwMDAgICAxLjM3MDAwMCAoICAxLjM3MzgwMCkKPiBmZmkgIDIu
MTYwMDAwICAgMC42NjAwMDAgICAyLjgyMDAwMCAoICAyLjgxODk2NikKPiBkbCAgIDMuNTAwMDAw
ICAgMS4wNjAwMDAgICA0LjU2MDAwMCAoICA0LjU1Mjc4OSkKPgoKVWJ1bnR1IDguMTAgNjRiaXQK
CiQgcnVieSAtLXZlcnNpb24gJiYgcnVieSBmZmlfYmVuY2gucmIKcnVieSAxLjguNyAoMjAwOC0w
OC0xMSBwYXRjaGxldmVsIDcyKSBbeDg2XzY0LWxpbnV4XQoxLjIuMy4zCjEuMi4zLjMKMS4yLjMu
MwogICAgICAgICB1c2VyICAgICBzeXN0ZW0gICAgICB0b3RhbCAgICAgICAgcmVhbApleHQgIDAu
MzIwMDAwICAgMC4wNzAwMDAgICAwLjM5MDAwMCAoICAwLjM5Njc3NCkKZmZpICAwLjc3MDAwMCAg
IDAuMTIwMDAwICAgMC44OTAwMDAgKCAgMC44OTUwOTMpCmRsICAgMi4wOTAwMDAgICAwLjI3MDAw
MCAgIDIuMzYwMDAwICggIDIuMzY1MDI5KQoKCgoKLS0gClJhZG9zs2F3IEJ1s2F0CgpodHRwOi8v
cmFkYXJlay5qb2dnZXIucGwgLSBt82ogYmxvZwo=

 

[Charles Oliver Nutter]

user system total real
ext 1.050000 0.320000 1.370000 ( 1.373800)
ffi 2.160000 0.660000 2.820000 ( 2.818966)
dl 3.500000 1.060000 4.560000 ( 4.552789)

All this using MacPorts MRI 1.8.7-p72 under OS X 10.5.5. The observed
overhead is slightly over 2x for ffi, probably not a big deal unless ffi
calls are used in tight loops I guess.

Seems like that pretty well seals the deal for ffi over dl, at the very
least. I'm also glad to see FFI wasn't even that bad, especially
considering it hasn't received any optimization.

- Charlie

 

[Charles Oliver Nutter]

This gives the following results:

1.2.3
1.2.3
1.2.3
user system total real
ext 1.050000 0.320000 1.370000 ( 1.373800)
ffi 2.160000 0.660000 2.820000 ( 2.818966)
dl 3.500000 1.060000 4.560000 ( 4.552789)

Ubuntu 8.10 64bit

$ ruby --version && ruby ffi_bench.rb
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
1.2.3.3
1.2.3.3
1.2.3.3
user system total real
ext 0.320000 0.070000 0.390000 ( 0.396774)
ffi 0.770000 0.120000 0.890000 ( 0.895093)
dl 2.090000 0.270000 2.360000 ( 2.365029)

Seems like about the same ratio...probably can be improved too!

- Charlie

 

[Charles Oliver Nutter]

Where can I file an issue? I have trouble to build in with ruby1.9 on 64bit.

Hop on the ruby-ffi project mailing lists, and I guess you can file a
bug in the tracker there too:

http://kenai.com/projects/ruby-ffi

- Charlie

 

[Sylvain Joyeux]

The JRuby team is proud to announce the release of FFI for Ruby 1.8.6/7
and 1.9!

FFI (gem install ffi) is a library for programmatically loading dynamic
libraries, binding functions within them, and calling those functions
from Ruby code. Here's a quick sample of binding and calling the getpid
C library function:

Interesting. On what kind of architectures is the binding part working ? I'm
using dyncall to do the actual interfacing work (http://www.dyncall.org/) in a
DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
What are you using on your side ?

Sylvain

 

[Charles Oliver Nutter]

Interesting. On what kind of architectures is the binding part working ? I'm
using dyncall to do the actual interfacing work (http://www.dyncall.org/) in a
DL-replacement library, but my problem is that dyncall does not like Linux-PPC.
What are you using on your side ?

Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not
certain about libffi specifically. but JNA claims to support OSX (ppc,
x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris
(x86, amd64, sparc, sparcv9) and Windows (x86, amd64).

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on. The
latter has been the case for several entries on the list; I myself was
the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time,
before which there was no shipped support.

There's certainly one way to find out...gem install ffi. Report back
here or on ruby-ffi mailing lists what you learn

http://kenai.com/projects/ruby-ffi

- Charlie

 

[jh+ruby-lang]

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on. The
latter has been the case for several entries on the list; I myself was
the build monkey for Solaris/AMD64 and Linux/AMD64 for a short time,
before which there was no shipped support.

There's certainly one way to find out...gem install ffi. Report back
here or on ruby-ffi mailing lists what you learn

Ubuntu 8.10 provides libffi on ppc. The gem builds, but a simple ruby
program hangs (rather uses 100% CPU).

I can investigate further (e.g building my own libffi), but that may
not happen immediately.

-jh

 

[Sylvain Joyeux]

Interesting. On what kind of architectures is the binding part working ? I'm

Ruby FFI uses libffi, as does JNA which ships with JRuby. I'm not
certain about libffi specifically. but JNA claims to support OSX (ppc,
x86, x86_64), linux (x86, amd64), FreeBSD/OpenBSD (x86, amd64), Solaris
(x86, amd64, sparc, sparcv9) and Windows (x86, amd64).

I did not know about libffi ... I'll have to look if I should not replace
dyncall by libffi then. Thanks for the info.

I'm not sure if linux-ppc support is not provided because it's not
supported or because nobody has a linux-ppc machine to build on.

I guess it is a bit of both. Given that libffi is supported on Linux/PPC64, I
guess having it on Linux/PPC should not be that much of a problem -- but still,
I think the calling convention can be slightly different between the two
architectures.

Sylvain

 

[Charles Oliver Nutter]

We (JRuby and Rubinius contributors) designed in the ability to
specify calling convention for a given library bound through FFI.
Largely this was needed to support Win32's stdcall, but if Linux/PPC
has a different convention and libffi supports it, then FFi will too
(though we may have to add it to the list of accepted conventions).

I'll check on the syntax we're supporting and get back to ya.

- Charlie (mobile)

On Nov 3, 2008, at 12:56, Sylvain Joyeux <[email protected]

 

[Charles Oliver Nutter]

Ubuntu 8.10 provides libffi on ppc. The gem builds, but a simple ruby
program hangs (rather uses 100% CPU).

I can investigate further (e.g building my own libffi), but that may
not happen immediately.

Hey, please do. Sounds like it *should* work, at least. I sure don't
have a PPC box to test on and I don't think anyone else involved in Ruby
FFI does either.

- Charlie