C ext / threading / callback issue, ruby-1.9.1p129

[reynhout]

Hi,

I've never written a C extension for Ruby before, and I'm struggling with
some demo code which uses C and/or Ruby pthreads, and a C or Ruby callback
(address passed as an arg into the C lib).

There's a watcher thread (in C) that waits for an event (the creation of
a file), then triggers whichever callback is specified.

Triggering the Ruby callback causes MRI to crash. The C callback works...
but when it completes, all Ruby threads appear to suspend (control is
never returned to Ruby / the threads are never rejoined?)

I'm guessing that the MRI crash is a dereferencing bug?? I haven't figured
that out obviously, but I have a feeling that the second problem (threads
never rejoin?) will show up after the crash is resolved.

I put the code up on github: http://github.com/reynhout/callme/ . There
is a bunch more description there, too.

There are two versions, C_ext (described above), and FFI_lib which uses
ruby-ffi to achieve the same goals ... but is also failing similarly.
I worked on the FFI_lib one first, then backed down to a straight C ext
trying to simplify it and make the bugs easier to find.

I'm pretty sure that if I can get the C_ext version working, the FFI_lib
one will fall into place. The ultimate goal is a Ruby library for Apache
ZooKeeper. The rest of the code is going great, but many ZK functions
depend on this sort of mechanism..

I hope the bugs are obvious to anyone who has written a C ext before.
I'd greatly appreciate any ideas or pointers (no pun intended ..

http://github.com/reynhout/callme/

Thank you!
Andrew

 

[Roger Pack]

There's a watcher thread (in C) that waits for an event (the creation of
a file), then triggers whichever callback is specified.

Hmm. MRI isn't "used" to other threads running--are you running 1.8.6
or something else?
=r

 

[reynhout]

Hmm. MRI isn't "used" to other threads running--are you running 1.8.6
or something else?

I'm using 1.9.1p129, but I tried the FFI version under 1.8.6 too, and
got similar MRI crashes and/or stuck threads. I did wrap the callback
invocation inside rb_thread_blocking_region, though I'm not sure it's
necessary for this sample code.

I haven't written a 1.8.6 version of the C ext (the syntax/semantics
are slightly different than 1.9), but I'd be very surprised to see
anything change -- these failures are consistent with MRI through
multiple versions and approaches.

This code DOES work correctly under JRuby. JRuby has a different
threading model, of course, but I don't understand it well enough to
determine why it fails under MRI.

Sample code and details at: http://github.com/reynhout/callme/

Thanks again,
Andrew

 

[Roger Pack]

I'm using 1.9.1p129, but I tried the FFI version under 1.8.6 too, and
got similar MRI crashes and/or stuck threads. I did wrap the callback
invocation inside rb_thread_blocking_region, though I'm not sure it's
necessary for this sample code.

Does the callback use ruby code from within the thread blocking region?
=r

 

[Andrew Reynhout]

Does the callback use ruby code from within the thread blocking region?

Yes, e.g.:

rb_thread_blocking_region((rb_blocking_function_t *)w->callback, tmp_x,
RUBY_UBF_IO, 0);

w->callback is the address of the Ruby proc. The address is correct
afaict, but note that this does crash MRI before the callback appears to
execute (dereferencing failure?). When w->callback is the address of
the C function, the callback runs, but the Ruby threads apparently
suspend.

Both callbacks in this sample code are very simple -- they just print
the int value of tmp_x, which is the number of seconds since the watcher
thread was started.

Thanks,
Andrew

 

[Luis Lavena]

Yes, e.g.:

rb_thread_blocking_region((rb_blocking_function_t *)w->callback, tmp_x,
RUBY_UBF_IO, 0);

w->callback is the address of the Ruby proc.  The address is correct
afaict, but note that this does crash MRI before the callback appears to
execute (dereferencing failure?).  When w->callback is the address of
the C function, the callback runs, but the Ruby threads apparently
suspend.

Both callbacks in this sample code are very simple -- they just print
the int value of tmp_x, which is the number of seconds since the watcher
thread was started.

Ruby-FFI issues, more related to callbacks can be asked at Ruby-ffi
mailing list:

http://kenai.com/projects/ruby-ffi/lists

If memory plays well, I saw something related to callbacks not long
ago there:

http://kenai.com/projects/ruby-ffi/lists/dev/archive

Sometimes the API hasn't been updated across the implementations.

 

[Andrew Reynhout]

Ruby-FFI issues, more related to callbacks can be asked at Ruby-ffi
mailing list:

Sorry, I didn't mean to confuse the issue by mentioning FFI. I did try
it with FFI initially (and have been discussing the problem on the
ruby-ffi list), but I've completely removed FFI from the current version
of the code. It's a straight C ext with no dependencies except gcc,
MRI, and mkmf.

Thanks,
Andrew

 

[Roger Pack]

rb_thread_blocking_region((rb_blocking_function_t *)w->callback, tmp_x,
RUBY_UBF_IO, 0);

Hmm. This question might be more easily answered by the core folk, but
perhaps you're assigning the value of w->callback to a ruby proc object,
when what you wanted was perhaps a pointer to a C method.

So the question is how to call proc objects in C?
I've never done it, but glancing at README.EXT appears to be something
like rb_iterate or rb_block_call (the latter for 1.9 esp.)

Not sure of that's right but it's about as far as my know-how goes.
GL.
=r