thread gurus please help...

A

Ara.T.Howard

following is a _simulation_ of a rather complex gui i'm working on. the gui
run a procedure which, itself, is communication with many other processes via
Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui
to update and do all the io (select/read, etc). this code seems to work fine
but i am very conerned that something might bite me later... if anyone would
care to comment on the interactions between the io and threads and if this is
a reccomended approach i'd love to hear it... remember this code might look a
bit funny but it demonstrative of what my code actually does. thanks in
advance for any help/advice

==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
require 'tk'
require 'open3'
require 'io/wait'
require 'io/nonblock'
require 'thread'
$VERBOSE=nil
STDOUT.sync = STDERR.sync = true
Thread.abort_on_exception = true

threads = []

#setup gui
root = TkRoot.new
label_a=TkLabel.new root, :text=>'label_a'
label_b=TkLabel.new root, :text=>'label_b'
label_a.pack
label_b.pack

# start the gui running
threads <<
Thread.new{ Tk.mainloop }

# simulation of my processing loop - which is communcating with another
# process and updating a widget with it's output
def update label, program
command="ruby -e '#{ program }'"
i,o,e = Open3::popen3 command
i.close

# we run in a thread so entire process is not blocked on io
# we DO need to wait for it to complete though, because in the real code i
# depend on values which are returned...
Thread.new do
o_done = false
e_done = false
loop do
rios, = select [o, e], nil, nil
rios.map do |rio|
break if o_done and e_done
if rio.eof?
rio == o ? o_done = true : e_done = true
next
end
# i like this better, but it hangs
# putting a critical section results in deadlock
# a semaphore is too slow...
#rio.nonblock{ text = rio.read }

# less elegant but works...
text = ''
sleep(0.42) and Thread.pass until rio.ready?
text << rio.getc while rio.ready?
label.configure :text=>text
end
end
end.join
end


# start background processing which will update the labels
threads <<
Thread.new{
update label_a, '4.times{p Time.now; sleep 1}'
}
threads <<
Thread.new{
update label_b, '4.times{|i| p "foobar_#{ i }"; sleep 1}'
}

# wait for everyone to finish
#threads.map{|t| t.join}
sleep
==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================
 
A

Ara.T.Howard

On Wed, 25 Feb 2004, Ara.T.Howard wrote:

forgot to mention that i have $VERBOSE=nil so i don't see the

"...fork terminates thread..."

err msg. i assume that, in this case (forking done in open3) it is OK if that
thread is terminated in the child - it obviously seem to continue running in
the parent...

-a
following is a _simulation_ of a rather complex gui i'm working on. the gui
run a procedure which, itself, is communication with many other processes via
Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui
to update and do all the io (select/read, etc). this code seems to work fine
but i am very conerned that something might bite me later... if anyone would
care to comment on the interactions between the io and threads and if this is
a reccomended approach i'd love to hear it... remember this code might look a
bit funny but it demonstrative of what my code actually does. thanks in
advance for any help/advice

==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====
require 'tk'
require 'open3'
require 'io/wait'
require 'io/nonblock'
require 'thread'
$VERBOSE=nil
STDOUT.sync = STDERR.sync = true
Thread.abort_on_exception = true

threads = []

#setup gui
root = TkRoot.new
label_a=TkLabel.new root, :text=>'label_a'
label_b=TkLabel.new root, :text=>'label_b'
label_a.pack
label_b.pack

# start the gui running
threads <<
Thread.new{ Tk.mainloop }

# simulation of my processing loop - which is communcating with another
# process and updating a widget with it's output
def update label, program
command="ruby -e '#{ program }'"
i,o,e = Open3::popen3 command
i.close

# we run in a thread so entire process is not blocked on io
# we DO need to wait for it to complete though, because in the real code i
# depend on values which are returned...
Thread.new do
o_done = false
e_done = false
loop do
rios, = select [o, e], nil, nil
rios.map do |rio|
break if o_done and e_done
if rio.eof?
rio == o ? o_done = true : e_done = true
next
end
# i like this better, but it hangs
# putting a critical section results in deadlock
# a semaphore is too slow...
#rio.nonblock{ text = rio.read }

# less elegant but works...
text = ''
sleep(0.42) and Thread.pass until rio.ready?
text << rio.getc while rio.ready?
label.configure :text=>text
end
end
end.join
end


# start background processing which will update the labels
threads <<
Thread.new{
update label_a, '4.times{p Time.now; sleep 1}'
}
threads <<
Thread.new{
update label_b, '4.times{|i| p "foobar_#{ i }"; sleep 1}'
}

# wait for everyone to finish
#threads.map{|t| t.join}
sleep
==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ==== ====

-a

--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================
 
N

nobu.nokada

Hi,

At Thu, 26 Feb 2004 09:04:47 +0900,
Ara.T.Howard wrote in [ruby-talk:93709]:
following is a _simulation_ of a rather complex gui i'm working on. the gui
run a procedure which, itself, is communication with many other processes via
Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui
to update and do all the io (select/read, etc). this code seems to work fine
but i am very conerned that something might bite me later... if anyone would
care to comment on the interactions between the io and threads and if this is
a reccomended approach i'd love to hear it... remember this code might look a
bit funny but it demonstrative of what my code actually does. thanks in
advance for any help/advice

In [ruby-dev:22861], Akira Tanaka supposed use of sysread
instead of non-blocking mode.

# As for blocking on non-blocking IO, I'll investigate later.
 
J

Joel VanderWerf

Ara.T.Howard said:
On Wed, 25 Feb 2004, Ara.T.Howard wrote:

forgot to mention that i have $VERBOSE=nil so i don't see the

"...fork terminates thread..."

err msg. i assume that, in this case (forking done in open3) it is OK if that
thread is terminated in the child - it obviously seem to continue running in
the parent...

I've never seen forking have an adverse effect on threads in the parent.
But there is a case where terminating all but one thread, which is what
happens in the child, has an adverse effect.

require 'thread'

m = Mutex.new

Thread.new { m.synchronize { sleep 1 } }

fork do
m.synchronize { puts "Didn't get here." }
end

m.synchronize { puts "Got here." }

Process.wait

This outputs the warning you saw and "Got here.".

I've gotten around this by explicitly removing the dead threads that
hold a mutex or are waiting for it. (Code available, if anyone's
interested.)

But that doesn't look like your problem, since the child is firing up a
whole new ruby.
 
A

Ara.T.Howard

I've never seen forking have an adverse effect on threads in the parent.
But there is a case where terminating all but one thread, which is what
happens in the child, has an adverse effect.

require 'thread'

m = Mutex.new

Thread.new { m.synchronize { sleep 1 } }

fork do
m.synchronize { puts "Didn't get here." }
end

m.synchronize { puts "Got here." }

Process.wait

This outputs the warning you saw and "Got here.".

wow. that is really bizarre. if i understand correctly all but the 'main'
thread is killed so the thread that is sleeping with the lock sort vaporizes
and leaves the mutex in an inconsistent (or locked at least) state?
I've gotten around this by explicitly removing the dead threads that hold a
mutex or are waiting for it. (Code available, if anyone's interested.)

you are braver than i ;-)
But that doesn't look like your problem, since the child is firing up a
whole new ruby.

sort of, the actual fork in my code are many satelite image processing steps
and it is done from within popen3 prior to an exec so i should be _ok_.

thanks for the reply - i'm feeling alot more confident now.

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================
 
A

Ara.T.Howard

Hi,

At Thu, 26 Feb 2004 09:04:47 +0900,
Ara.T.Howard wrote in [ruby-talk:93709]:
following is a _simulation_ of a rather complex gui i'm working on. the gui
run a procedure which, itself, is communication with many other processes via
Open3::popen3 (i _need_ the stderr)... anyhow, i'm having fits getting the gui
to update and do all the io (select/read, etc). this code seems to work fine
but i am very conerned that something might bite me later... if anyone would
care to comment on the interactions between the io and threads and if this is
a reccomended approach i'd love to hear it... remember this code might look a
bit funny but it demonstrative of what my code actually does. thanks in
advance for any help/advice

In [ruby-dev:22861], Akira Tanaka supposed use of sysread instead of
non-blocking mode.

wow. tried babelfish but i'm a bit lost on that one ;-)

what is a 'bean jam ball'? sounds like something good...

i tried replacing with sysread but am now failing with

sysread for buffered IO (IOError)

....

for now i seem to be having luck with (this is after a select because i am
multiplexing stdout and stderr from another process)

text = ''
sleep(0.042) and Thread.pass until rio.ready?
text << rio.getc while rio.ready?
label.configure :text=>text

but reading only a character at a time when the io object is ready i hope to
never block. this is what i had hoped to accomplish by

io.nonblock{ io.read }

but this does seem to work fine - i don't really know how to say 'read
everything that is available but do not block' from within a multi-threaded
application...
# As for blocking on non-blocking IO, I'll investigate later.

thanks. i'd be quite interested to hear what you find.

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL :: http://www.ngdc.noaa.gov/stp/
| TRY :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done
===============================================================================
 
T

Tanaka Akira

Ara.T.Howard said:
i tried replacing with sysread but am now failing with

sysread for buffered IO (IOError)

You use eof? or other stdio related methods.
Fortunately, you don't need such methods in this case.
io.nonblock{ io.read }

but this does seem to work fine - i don't really know how to say 'read
everything that is available but do not block' from within a multi-threaded
application...

You don't need nonblock because sysread doesn't block if some data are
available. Note that IO.select blocks until some data are available.

For example, your [ruby-talk:93618] can be modified as follows.

require 'open3'
STDOUT.sync = STDERR.sync = true

command = <<'End'
ruby -e "
STDOUT.sync = true
10.times {
STDOUT.puts Time.now; sleep 1
STDERR.puts Time.now; sleep 1
}"
End
i,o,e = Open3::popen3 command
i.close

inputs = [o, e]
p inputs
t = Thread.new do
until inputs.empty?
rios, = select inputs
rios.each do |rio|
p 'start...'
begin
text = rio.sysread(8192)
rescue EOFError
inputs.delete rio
next
end
p [rio, text]
p 'finish'
end
end
end

t.join

Also, IO.select can be removed with two threads.

require 'open3'
STDOUT.sync = STDERR.sync = true

command = <<'End'
ruby -e "
STDOUT.sync = true
10.times {
STDOUT.puts Time.now; sleep 1
STDERR.puts Time.now; sleep 1
}"
End
i,o,e = Open3::popen3 command
i.close

t1 = Thread.new do
loop {
begin
text = o.sysread(8192)
rescue EOFError
break
end
puts "STDOUT: #{text.inspect}"
}
end
t2 = Thread.new do
loop {
begin
text = e.sysread(8192)
rescue EOFError
break
end
puts "STDERR: #{text.inspect}"
}
end

t1.join
t2.join

Note that sysread may block in this case when no data is available.
But I think it is not a problem because you cannot do any computation
on the no data. I think it is even good behaviour because it avoids
busy loop.
 
N

nobu.nokada

Hi,

At Thu, 26 Feb 2004 13:54:48 +0900,
Ara.T.Howard wrote in [ruby-talk:93721]:
what is a 'bean jam ball'? sounds like something good...

'Bean jam' is sweet paste made boiling red beans and sugar. It
looks black in common, but there is white one and others. And
Japanese are in the habit of eating rice balls with bean jam in
each equinox.
i tried replacing with sysread but am now failing with

sysread for buffered IO (IOError)

See [ruby-talk:93722] about sysread.

Anyway, I feel non-blocking IO should never block in this case,
simpler code which can reproduce your issue.

require 'io/nonblock'

unless rio = IO.popen("-")
loop do
p Time.now
sleep 1
end
end

Thread.new do
until rio.eof?
#Thread.critical = true # blocks w/o this
text = rio.nonblock{ rio.read }
#Thread.critical = false # blocks w/o this
print text
end
end.join


Index: io.c
===================================================================
RCS file: /cvs/ruby/src/ruby/io.c,v
retrieving revision 1.261
diff -u -2 -p -d -r1.261 io.c
--- io.c 25 Feb 2004 12:17:39 -0000 1.261
+++ io.c 26 Feb 2004 06:38:59 -0000
@@ -889,4 +889,10 @@ rb_io_to_io(io)
}

+#ifndef O_NONBLOCK
+# ifdef O_NDELAY
+# define O_NONBLOCK O_NDELAY
+# endif
+#endif
+
/* reading functions */
long
@@ -903,4 +909,8 @@ rb_io_fread(ptr, len, f)
long i = READ_DATA_PENDING_COUNT(f);
if (i <= 0) {
+#if defined(HAVE_FCNTL) && defined(F_GETFL) && defined(O_NONBLOCK)
+ i = fcntl(fileno(f), F_GETFL);
+ if (i == -1 || (i & O_NONBLOCK)) break;
+#endif
rb_thread_wait_fd(fileno(f));
i = READ_DATA_PENDING_COUNT(f);
@@ -5508,10 +5518,6 @@ Init_IO()
rb_file_const("CREAT", INT2FIX(O_CREAT));
rb_file_const("EXCL", INT2FIX(O_EXCL));
-#if defined(O_NDELAY) || defined(O_NONBLOCK)
-# ifdef O_NONBLOCK
+#ifdef O_NONBLOCK
rb_file_const("NONBLOCK", INT2FIX(O_NONBLOCK));
-# else
- rb_file_const("NONBLOCK", INT2FIX(O_NDELAY));
-# endif
#endif
rb_file_const("TRUNC", INT2FIX(O_TRUNC));
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,579
Members
45,053
Latest member
BrodieSola

Latest Threads

Top