Problem with concurrency

[Tasos Laskos]

Hi guys,

I'm having an issue with a multi-threaded system I'm developing.

Sorry if my question sounds too generic and unspecific but:
Are there any known pitfalls that one might come across when using
threads?

This is the code that runs the threads:
(I'm new to this so the code is borrowed from another post from this
list.)
---------
def run_mods( page_data, structure )

mod_queue = Queue.new

@threads = ( 1..@opts[:threads] ).map {
|i|
Thread.new( mod_queue ) {
|q|
until( q == ( curr_mod = q.deq ) )

if( !run_module?( curr_mod , structure ) )
print_verbose( 'Skipping ' + curr_mod.to_s +
', nothing to audit.' )
next
end

print_debug( )
print_debug( 'Thread-' + i.to_s + " " +
curr_mod.inspect )
print_debug( )

print_status( curr_mod.to_s )

mod_new = curr_mod.new( page_data, structure )

mod_new.prepare if curr_mod.method_defined?(
'prepare' )
mod_new.run
mod_new.clean_up if curr_mod.method_defined?(
'clean_up' )

while( handle_interrupt( ) )
end

end
}
}

# enque the loaded mods
for mod in ls_loaded_mods
mod_queue.enq mod
end

@threads.size.times { mod_queue.enq mod_queue }
@threads.each { |t| t.join }

end
-----------------

The modules call their parent and the parent uses Net::HTTP.
Then they get the HTTP response via the parent and manipulate it in some
way.
My main problem is that from what I've gathered some HTTP calls fail
without an error.

Some modules just don't produce any results when the thread count is set
higher than 1.

I normally wouldn't bother you with something so broad but I've been
debugging this for 24 hours straight so I figured that someone else
might have run to a similar problem and save me 24 more hours of
debugging.

Has anyone else had a similar experience?

 

[Robert Klemme]

Hi guys,

I'm having an issue with a multi-threaded system I'm developing.

Sorry if my question sounds too generic and unspecific but:
Are there any known pitfalls that one might come across when using
threads?

This is the code that runs the threads:
(I'm new to this so the code is borrowed from another post from this
list.)
---------
def run_mods( page_data, structure )

mod_queue = Queue.new

@threads = ( 1..@opts[:threads] ).map {
|i|
Thread.new( mod_queue ) {
|q|
until( q == ( curr_mod = q.deq ) )

if( !run_module?( curr_mod , structure ) )
print_verbose( 'Skipping ' + curr_mod.to_s +
', nothing to audit.' )
next
end

print_debug( )
print_debug( 'Thread-' + i.to_s + " " +
curr_mod.inspect )
print_debug( )

print_status( curr_mod.to_s )

mod_new = curr_mod.new( page_data, structure )

mod_new.prepare if curr_mod.method_defined?(
'prepare' )
mod_new.run
mod_new.clean_up if curr_mod.method_defined?(
'clean_up' )

while( handle_interrupt( ) )
end

end
}
}

# enque the loaded mods
for mod in ls_loaded_mods
mod_queue.enq mod
end

@threads.size.times { mod_queue.enq mod_queue }
@threads.each { |t| t.join }

end
-----------------

The modules call their parent and the parent uses Net::HTTP.
Then they get the HTTP response via the parent and manipulate it in some
way.
My main problem is that from what I've gathered some HTTP calls fail
without an error.

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Kind regards

robert

 

[Tasos Laskos]

Try this at the top of your program:

Thread.abort_on_exception = true

Then you will probably see errors.

Kind regards

robert

Hi Robert,

Thanks for the reply, however I didn't see any errors.

This is going to be a bitch to debug...

 

[Tasos Laskos]

Hi Robert,

Thanks for the reply, however I didn't see any errors.

This is going to be a bitch to debug...

Huh...it seems that the problem was actually caused by the localhost
webserver.
My laptop had a crazy uptime and all its memory (virtual and ram) had
been somewhat exhausted.

After a reboot everything started to work fine.

I can't believe all those hours of debugging went to waist.

 

[Robert Klemme]

Huh...it seems that the problem was actually caused by the localhost
webserver.
My laptop had a crazy uptime and all its memory (virtual and ram) had
been somewhat exhausted.

After a reboot everything started to work fine.

I can't believe all those hours of debugging went to waist.

If it can happen once chances are that it will happen again. It's
probably worth to dig a bit and try to find out what caused the memory
issue in the first place. Maybe you have a leaky application. What OS btw?

Kind regards

robert

 

[Tasos Laskos]

If it can happen once chances are that it will happen again. It's
probably worth to dig a bit and try to find out what caused the memory
issue in the first place. Maybe you have a leaky application. What OS
btw?

Kind regards

robert

Well, the laptop was on for 2-3 days constantly running fullscreen flash
video, eclipse, apache2 & mysql ( being constantly hammered by Arachni
due to the debugging itself) and around 13 tabs of Firefox spread over
several windows.

So in hindsight it's not much of a surprise.

I'm running the latest Kubuntu with Xfce, so I guess it has now become
Xubuntu...