Problems with cross-thread violation

[Gin Mendi]

Been having problems with cross-thread violation in my program and been
researching on how I can get around it. I really want to continue using
ruby, because learning and using it has been fun, but if I can't find
some solution I guess I have to drop it for now.

Basically I'm using a dll provided by a vendor to access the server to
get a live feed of data. I made a module using Ruby/DL to access the
function calls like logging on to the server and resquesting a snapshot
data or a session of a live feed of data. Everything has been working
great except my attempts to establish a live feed.

The live feed requires a callback function where the server will call
whenever new data is available. This is where I encounter cross-thread
violations.

I tried simple implementations of requesting a live feed and supplying a
callback function using Ruby/DL. I then just have an infinite loop to
keep the programming waiting. The callback is performed but after a
certain number of calls I get the cross-thread violation on
rb_threa_schedule() error.

From what I read is this because the callback is initiated in another
thread apart from the ruby's main thread? Even if this is the cause, I
still have no idea how I can remedy this. I tried using EventMachine
hoping that I could pass the pointer from the callback to some spawned
process but I eventually got the cross-thread violation.

Is there an approach I can use to solve this problem or do I just simply
give up and use something else. I'm hoping some of the gurus here can
give me some information on what I can do.

 

[Eleanor McHugh]

Is there an approach I can use to solve this problem or do I just
simply
give up and use something else. I'm hoping some of the gurus here can
give me some information on what I can do.

Do you have code available that we can study?


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

 

[Gin Mendi]

Do you have code available that we can study?

Thanks again for taking the time to read this!

require 'feed_library'

# my callback for the live feed
FEED_CALLBACK = DL.callback('0PL') { |result_list, count| }
if result_list
record = FeedRecord.new(result_list)
puts record.code
end
}

# handle used to request for a feed
handle = FeedLibrary::log_on("user", "password")

session = get_live_feed(handle, FEED_CALLBACK)

# my attempt to keep this running and waiting on the callback:
while true

end

Here's one I tried. Been playing around with different approaches. With
this method I noticed I always get the cross-thread violation when I use
the FeedRecord structure. If I don't use the structure and do something
else like using the struct! method I'm able to get away with the
callback being performed a couple of times then it eventually leads to a
cross-thread violation.

 

[Eleanor McHugh]

Thanks again for taking the time to read this!

require 'feed_library'

# my callback for the live feed
FEED_CALLBACK = DL.callback('0PL') { |result_list, count| }
if result_list
record = FeedRecord.new(result_list)
puts record.code
end
}

# handle used to request for a feed
handle = FeedLibrary::log_on("user", "password")

session = get_live_feed(handle, FEED_CALLBACK)

# my attempt to keep this running and waiting on the callback:
while true

end

No need to be uncivilised Idiomatic Ruby for an endless loop is
better expressed as:

loop do
end

although that shouldn't make any difference to your program.

Here's one I tried. Been playing around with different approaches.
With
this method I noticed I always get the cross-thread violation when I
use
the FeedRecord structure. If I don't use the structure and do
something
else like using the struct! method I'm able to get away with the
callback being performed a couple of times then it eventually leads
to a
cross-thread violation.

Does get_live_feed use a separate thread internally? If so the cross-
thread violation could well be related to garbage collection of the
callback in which case you may need a separate callback instance for
each feed handle. Also I note that you're using a constant to hold the
callback, have you tried using a variable instead? Try turning off
garbage collection altogether and see whether that affects the errors
you receive. I've had some very amusing results doing that with DL
callbacks


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

 

[Gin Mendi]

No need to be uncivilised Idiomatic Ruby for an endless loop is
better expressed as:

loop do
end

sweet thanks! Still trying to get a hang of ruby so tips like this are
great for me

Does get_live_feed use a separate thread internally? If so the cross-
thread violation could well be related to garbage collection of the
callback in which case you may need a separate callback instance for
each feed handle. Also I note that you're using a constant to hold the
callback, have you tried using a variable instead? Try turning off
garbage collection altogether and see whether that affects the errors
you receive. I've had some very amusing results doing that with DL
callbacks

I'm unsure if get_live_feed uses a seperate thread. I just have the dll
and no source. If get_live_feed does use a seperate thread how would I
make a seperate feed handle? Does that mean I need to declare a new
callback and pass that each time I call live feed?

I tried using a variable but I still get the cross-thread violation. I
noticed though if I do not use the structure and use other methods to
display the data in the pointer I get more calls out of the callback
before I hit the cross-thread violation. Just wondering was it wrong to
use a constant for the callback?

I did some research on garbage collection. Is turning off garbage
collection simply GC.disable? If I do disable garbage collection does
that mean I have to run the free method on my pointers or put some extra
code to free up memory?

Thanks again for all the help

 

[Eleanor McHugh]

sweet thanks! Still trying to get a hang of ruby so tips like this are
great for me

I've been using Ruby for eight years now and I still find new ways of
expressing myself - it's a rich language

I'm unsure if get_live_feed uses a seperate thread. I just have the
dll
and no source. If get_live_feed does use a seperate thread how would I
make a seperate feed handle? Does that mean I need to declare a new
callback and pass that each time I call live feed?

Not necessarily, but it's certainly an experiment I'd consider as part
of taming the problem. If you know C reasonably well I also recommend
taking a peak through /ext/dl in the Ruby sources. That's not for the
faint-hearted though.

I tried using a variable but I still get the cross-thread violation. I
noticed though if I do not use the structure and use other methods to
display the data in the pointer I get more calls out of the callback
before I hit the cross-thread violation. Just wondering was it wrong
to
use a constant for the callback?

It's best not to use a constant unless the object it refers to is
intended to last for the lifetime of the program. For one thing
constants in Ruby won't prevent assignment, nor are the objects they
refer to proof against state changes. If you really want constant
behaviour in some sense you need to freeze the object in question.

I'm uncertain whether this has any bearing on your particular callback
problem, but freezing the callback object would be yet another
diagnostic tool worth trying as any subsequent attempt to modify it
would raise an exception.

I did some research on garbage collection. Is turning off garbage
collection simply GC.disable? If I do disable garbage collection does
that mean I have to run the free method on my pointers or put some
extra
code to free up memory?

No, it just means that while garbage collection is disabled your
unreferenced objects will remain in memory. Once you reenable it all
those dead objects will then be dealt with during the next collection
cycle, including freeing up any memory associated with DataPtr objects
you may have created with DL::malloc.

Thanks again for all the help

That's what we're here for


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

 

[Gin Mendi]

Not necessarily, but it's certainly an experiment I'd consider as part
of taming the problem. If you know C reasonably well I also recommend
taking a peak through /ext/dl in the Ruby sources. That's not for the
faint-hearted though.

I'm only learning C now as I was trying to look at the source code. At
first glance I couldn't understand anything there but I'll study it if I
run out of ideas. Anything in particular that I should study?

No, it just means that while garbage collection is disabled your
unreferenced objects will remain in memory. Once you reenable it all
those dead objects will then be dealt with during the next collection
cycle, including freeing up any memory associated with DataPtr objects
you may have created with DL::malloc.

what about the pointers in the callback will those remain unreferrenced
or are they automatically freed when the callback ends?

Well I tried disabling GC as you suggested but I got the same result.
Here's what I did:

require 'feed_library'

module live_feed

# my callback for the live feed
fc = DL.callback('0PL') { |result_list, count| }
if result_list
record = FeedRecord.new(result_list)
puts record.code
end
}

# handle used to request for a feed
handle = FeedLibrary::log_on("user", "password")

if handle
GC.disable

session = get_live_feed(handle, fc)

loop do
end

GC.enable
end

end

I still got the cross-thread violation on rb_thread_schedule()

I tried changing the callback. Instead of using the struct class
FeedRecord I instead used the struct! method. So I made it simple just
to access 3 fields from the data structure:

fc = DL.callback('0PL') { |result_list, count| }
if result_list
result_list.struct('C16FF', :code, :value, :stat)
puts result_list[:code].pack('C16')
puts result_list[:value]
end
}

I instead got a crash where I would see the printout of the callback
with mixed in letters forming segmentation fault. Other runs would just
flat out crash back to the command prompt.

 

[Eleanor McHugh]

I'm only learning C now as I was trying to look at the source code. At
first glance I couldn't understand anything there but I'll study it
if I
run out of ideas. Anything in particular that I should study?

If C is a new language to you then the Ruby sources are definitely not
the place to learn it.

what about the pointers in the callback will those remain
unreferrenced
or are they automatically freed when the callback ends?

They should also be freed. There are caveats and edge cases where this
might not be the case, but based on the code you've posted none of
those seem to apply.

Well I tried disabling GC as you suggested but I got the same result.
Here's what I did:

require 'feed_library'

module live_feed

# my callback for the live feed
fc = DL.callback('0PL') { |result_list, count| }
if result_list
record = FeedRecord.new(result_list)
puts record.code
end
}

# handle used to request for a feed
handle = FeedLibrary::log_on("user", "password")

Are you certain that in the case of no feed being available, you're
receiving a nil rather than a zero? Because if not then the following
condition will always be true and then there'll be times when you try
and access a non-existent feed.

if handle
GC.disable

session = get_live_feed(handle, fc)

loop do
end

GC.enable
end

end

I still got the cross-thread violation on rb_thread_schedule()

I tried changing the callback. Instead of using the struct class
FeedRecord I instead used the struct! method. So I made it simple just
to access 3 fields from the data structure:

fc = DL.callback('0PL') { |result_list, count| }
if result_list
result_list.struct('C16FF', :code, :value, :stat)
puts result_list[:code].pack('C16')
puts result_list[:value]
end
}

I instead got a crash where I would see the printout of the callback
with mixed in letters forming segmentation fault. Other runs would
just
flat out crash back to the command prompt.

You could try installing Tenderlove's Never Say Die library from
github, it allows you to rescue segfaults which might allow you to
extract some useful program state or else continue as if nothing had
gone wrong (but that's a really bad idea in the majority of cases).

You might also try coding your wrapper with ruby-ffi rather than ruby/
dl.


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

 

[Gin Mendi]

Are you certain that in the case of no feed being available, you're
receiving a nil rather than a zero? Because if not then the following
condition will always be true and then there'll be times when you try
and access a non-existent feed.

Yep handle will be nil if log_on fails or encounters an error.

You could try installing Tenderlove's Never Say Die library from
github, it allows you to rescue segfaults which might allow you to
extract some useful program state or else continue as if nothing had
gone wrong (but that's a really bad idea in the majority of cases).

You might also try coding your wrapper with ruby-ffi rather than ruby/
dl.

Ok I'll try looking and studying both of those suggestions. Might take a
while for me to redo with ffi but I'll post back here what the results
were.

You mentioned earlier that I may need a separate callback instance for
each feed handle:

Does get_live_feed use a separate thread internally? If so the cross-
thread violation could well be related to garbage collection of the
callback in which case you may need a separate callback instance for
each feed handle.

Sorry but I don't fully understand how I could go about doing this? How
would this work?

Also a little side problem I still haven't solved regarding my callback:

fc = DL.callback('0PL') { |result_list, count| }
if result_list
result_list.struct('C16FF', :code, :value, :stat)
puts result_list[:code].pack('C16')
puts result_list[:value]
end
}

you'll notice I have a count parameter which is the number of records in
result_list. I assume result list is an array if count > 1. Been trying
to figure out how to extract each record if count > 1. Right now only
managed to do this with just one record as you can see. How could I
convert my pointer to an array then get each element/record? I'm
assuming result_list is an array. I tried result_list.ptr.to_a but that
just made the program crash back to the command prompt with no error.

Anyway thanks again for all the help!