Executing system commands in threads under Ruby 1.8.6

V

vhaerun vh

I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn't seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby's thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?
 
J

Joel VanderWerf

vhaerun said:
I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn't seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby's thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?

What do you mean by external binary? Something you invoke with #system
or the like?

$ ruby -e 't=Thread.new {system "echo foo; sleep 1; echo bar"}; system
"echo baz; sleep 1; echo quux"; t.join'
foo
baz
bar
quux

Note the order of output: baz before bar. So it's not sequential.
 
R

Robert Klemme

2009/9/7 vhaerun vh said:
I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn't seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby's thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?

I don't think that is necessary. Can you provide a example of the
phenomenon you describe? Normally, external programs are separate
processes and work independently. It may be though that if you do the
IO handling for external processes not properly that it looks like
they are executed sequentially because they are blocked in IO
operations.

Kind regards

robert
 
B

Bertram Scharpf

Hi,

Am Montag, 07. Sep 2009, 20:06:32 +0900 schrieb Eleanor McHugh:
Backticks are blocking calls which return the entirety of their stdout as a
single string on completion of the subprocess.

I cannot open the above link. So I rewrite Joel's code snippet
using backticks:

t = Thread.new {
puts `echo foo; sleep 1; echo bar`
}
puts `echo baz; sleep 1; echo quux`
t.join

Stricly spoken it's not blocking. The "baz" is written to its pipe
before "bar". But it is stored until "quux" is echoed and puts is
asked to write it out.

Bertram
 
G

geo ssscripting

The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

`ping #{ip_addr}`

This happened from inside each individual thread. How could I replace
the backticks so that they won't block ?
 
R

Robert Klemme

2009/9/7 geo ssscripting said:
The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

`ping #{ip_addr}`

This happened from inside each individual thread. How could I replace
the backticks so that they won't block ?

You can use one of the popen methods, e.g.

IO.popen ["ping", ip_addr] do |io|
io.each {|l| puts l}
end

Example

15:37:19 ~$ ruby <<EOF
threads = (1..2).map do |i|
Thread.new i do |ii|
IO.popen 'ping 192.168.110.74' do |io|
io.each do |line|
printf "%2d %s", ii, line
end
end
end
end
threads.map do |th|
th.value
end
EOF
1
1 Pinging 192.168.110.74 with 32 bytes of data:
1
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Pinging 192.168.110.74 with 32 bytes of data:
2
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1
1 Ping statistics for 192.168.110.74:
1 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
1 Approximate round trip times in milli-seconds:
1 Minimum = 0ms, Maximum = 0ms, Average = 0ms
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Ping statistics for 192.168.110.74:
2 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
2 Approximate round trip times in milli-seconds:
2 Minimum = 0ms, Maximum = 0ms, Average = 0ms
15:37:38 ~$ ruby -version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]
-e:1: undefined local variable or method `rsion' for main:Object (NameError)
15:37:53 ~$


Cheers

robert
 
B

Bertram Scharpf

Hi,

Am Montag, 07. Sep 2009, 21:04:53 +0900 schrieb geo ssscripting:
The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

`ping #{ip_addr}`

This happened from inside each individual thread. How could I replace
the backticks so that they won't block ?

You could fork-exec-wait as described in
<http://www.ruby-doc.org/docs/ProgrammingRuby/html/ref_m_process.html#Process.waitpid>.

To retrieve the exit status call waitpid2 as described below
waitpid.

Just fill an array with pids and you don't even need threading.

Bertram
 
E

Eleanor McHugh

I cannot open the above link. So I rewrite Joel's code snippet
using backticks:

t = Thread.new {
puts `echo foo; sleep 1; echo bar`
}
puts `echo baz; sleep 1; echo quux`
t.join

Stricly spoken it's not blocking. The "baz" is written to its pipe
before "bar". But it is stored until "quux" is echoed and puts is
asked to write it out.


From Ruby's perspective the backtick is definitely a blocking IO
operation, just as it is in shell script.


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
 
G

geo ssscripting

Here's the rewrite using IO.popen:

threads = []

(1..254).each do |i|
puts "pinging #{i}"
threads << Thread.new {
content = ""
IO.popen("ping 192.168.0.#{i}") do |io|
io.each { |l| content << l }
end
content
}
end

threads.each do |t|
t.join
puts t.value
end

This doesn't behave different. My Ruby -v outputs this :

ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

I'm running Windows XP.
 
B

Bertram Scharpf

Hi,

Am Dienstag, 08. Sep 2009, 01:11:23 +0900 schrieb Eleanor McHugh:
From Ruby's perspective the backtick is definitely a blocking IO operation,
just as it is in shell script.

Yes, of course. I just examined the interpreter's source. The
waitpid function is called _without_ the WNOHANG flag. Before that
the child processes output is read and appended to a string until
the stream is closed what usually happens when the program
terminates.

What I meant was just that the child process is really running and
writing "foo" and "baz" to the pipes 1 second before the output
can by noticed in the parent.

Bertram
 
R

Robert Klemme

Here's the rewrite using IO.popen:

threads = []

(1..254).each do |i|
puts "pinging #{i}"
threads << Thread.new {
content = ""
IO.popen("ping 192.168.0.#{i}") do |io|
io.each { |l| content << l }
end
content
}
end

threads.each do |t|
t.join

#join is superfluous when using #value.
puts t.value
end

This doesn't behave different. My Ruby -v outputs this :

What exactly do you mean? What do you expect? If you refer to seeing
the output of both ping commands sequentially: with the code you
presented you always will get your output sequentially simply because
you wait until all threads finish and then you'll iterate them and print
the output of one thread at a time. Your ping commands will run in
parallel.
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

I'm running Windows XP.

robert
 
G

geo ssscripting

The first 4 threads or so are created almost instantly, and after that,
everything runs as if only one thing is executed at a time. I get the
same behaviour no matter how many times I try to run it.
 
B

Bertram Scharpf

Hi,

Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram Scharpf:
What I meant was just that the child process is really running and
writing "foo" and "baz" to the pipes 1 second before the output
can by noticed in the parent.

In other words: the child processes don't block each other.
Is that ok?

Bertram
 
E

Eleanor McHugh

Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram Scharpf:

In other words: the child processes don't block each other.
Is that ok?

Rather let's say that child processes won't block each other due to
the underlying implementation of backtick. It is however still
possible to introduce blocking behaviour by using operations in those
children which would cause blocking, such as accessing a semaphore or
other blocking system resource.


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,439
Members
44,829
Latest member
PIXThurman

Latest Threads

Top