Ruby redirection woes

  • Thread starter Steven Kah Hien Wong
  • Start date
S

Steven Kah Hien Wong

----=_NextPart_ST_12_56_33_Tuesday_October_26_2004_13396
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi,

I have made a class, called CommandRunner, that I have been experiencing
some unusual behaviour with. I have lived with it for around two months
now, hoping I will stumble across a solution eventually. But alas, no.
:( I designed the class to provide the following functions:

- runs other programs in a child process (with exec()),
- allows each command-line argument to be explicitly passed in as
individual arguments of the CommandRunner's constructor, and
- provides IO streams to the parent for the child's standard out,
standard error, and standard in streams.

As far as I know, the existing available functions (such as Kernel.popen
and Kernel.system) can provide these, but one that does them all does
not exist. For instance, popen gives you only one IO stream for reading
the standard in AND error, and writing standard in; not seperate IO
streams for standard in and error. It also doesn't let you specify
command-line arguments seperately (according to the documentation,
atleast). If one of your arguments have spaces, then I think you will
need to quote it or something - I'm not sure.

Basically, when I run it and redirect output from the script to the
shell, I get additional output from the child process being run by
CommandRunner, when it should not be there.

To better explain, run the command_runner_shell_test.rb (file attached)
as follows, to see the wackiness:

$ ruby command_runner_shell_test.rb > output.txt
$ cat output.txt
Hello, world! 2 times!
Hello, world! 2 times!
Hello, world! 2 times!

It should only print two times...

Increase the loop in command_runner_shell_test.rb to five times, and
watch the excess prints grow (what seems to be roughly) exponentially.

Uncomment the second block in the command_runner_shell_test.rb script,
and see it all get jumbled. More wackiness. o_O

All the weird thread and waiting for the 'R' character, trickery is so
that I can guaranty that by the time I am calling the waitpid function,
the process and PID still exist. In other words, it is possible for the
child process to be so quick that it finishes executing before waitpid()
gets called). So on a sidenote, does anyone know a cleaner way of
performing interprocess communication? Like how do I tell the forked,
child process that the parent is ready and waiting.

Another problem is sometimes it just hangs at the 'exec' line. I see the
process and command line being run from 'ps aux', but it doesn't seem to
ever exit. But one problem at a time. I suspect this one might be linked
with the above. Something to do with the file streams not being ready, I
think?

Take a look, and have fun figuring it out. :p

Regards,

Steven
--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E (e-mail address removed)
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.
----=_NextPart_ST_12_56_33_Tuesday_October_26_2004_13396
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="command_runner.rb"

class CommandRunner
attr :command
attr :childPid

def initialize(*command)
@command = command
@childPid = nil
@waitingThread = nil

@readPipe = nil
@readErrorPipe = nil
@writePipe = nil
end

def closeWrite
@writePipe.close
end

def kill
Process.kill("KILL", @childPid)
end

def read
return @readPipe.read
end

def readError
return @readErrorPipe.read
end

def run
parent_to_child_read, parent_to_child_write = IO.pipe
child_to_parent_read, child_to_parent_write = IO.pipe
child_to_parent_error_read, child_to_parent_error_write = IO.pipe

@childPid = fork do
parent_to_child_write.close
child_to_parent_read.close
child_to_parent_error_read.close

$stdin.reopen(parent_to_child_read) or
raise "Unable to redirect STDIN"
$stdout.reopen(child_to_parent_write) or
raise "Unable to redirect STDOUT"
$stderr.reopen(child_to_parent_error_write) or
raise "Unable to redirect STDERR"

# Wait until parent is ready before we start doing anything
if parent_to_child_read.readchar.chr != "R"
raise "Unexpected input from parent"
end

exec(*@command)
end

@waitingThread = Thread.new do
return_code = -1

begin
return_code = Process.waitpid2(@childPid)
rescue SystemError
raise "Process finished running already!"
end
return_code = return_code[1].exitstatus

return_code
end

child_to_parent_write.close
child_to_parent_error_write.close
parent_to_child_read.close

@readPipe = child_to_parent_read
@readErrorPipe = child_to_parent_error_read
@writePipe = parent_to_child_write

# Tell child we are ready
@writePipe.write("R")
@writePipe.flush
end

#--------------------------------------------------------------------------
# Description: Waits for command to exit
# Returns : The return code of the program when it exited
#--------------------------------------------------------------------------
def wait
if not @childPid or not @waitingThread
raise "Waiting for a process that has not started"
end

return_value = @waitingThread.value

@waitingThread = nil
@childPid = nil

return return_value
end

def write(string)
@writePipe.write(string)
end
end

----=_NextPart_ST_12_56_33_Tuesday_October_26_2004_13396
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="command_runner_shell_test.rb"

require 'command_runner'

2.times do
command_runner = CommandRunner.new("/bin/echo", "Hello, world! 2 times!")
command_runner.run
puts command_runner.read
end

# Uncomment the following as well for more wackiness

# 3.times do
# command_runner = CommandRunner.new("/bin/echo", "Hello, world! 3 times!")
# command_runner.run
# puts command_runner.read
# end

----=_NextPart_ST_12_56_33_Tuesday_October_26_2004_13396--
 
S

Steven Kah Hien Wong

Hi,

I have made a class, called CommandRunner, that I have been experiencing
some unusual behaviour with. I have lived with it for around two months
now, hoping I will stumble across a solution eventually. But alas, no.
:( I designed the class to provide the following functions:

Since I haven't got any responses yet. Should I forward this to
ruby-core? Or is the explanation too lengthy?

Steven
--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E (e-mail address removed)
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.
 
Y

Yukihiro Matsumoto

Hi,

In message "Re: Ruby redirection woes"

|On Tue, Oct 26, 2004 at 01:56:38PM +0900, Steven Kah Hien Wong wrote:
|> Hi,
|>
|> I have made a class, called CommandRunner, that I have been experiencing
|> some unusual behaviour with. I have lived with it for around two months
|> now, hoping I will stumble across a solution eventually. But alas, no.
|> :( I designed the class to provide the following functions:
|
|Since I haven't got any responses yet. Should I forward this to
|ruby-core? Or is the explanation too lengthy?

Somehow I missed your first mail. I will check it later.
Stay tuned.

matz.
 
B

Brian Candler

To better explain, run the command_runner_shell_test.rb (file attached)
as follows, to see the wackiness:

$ ruby command_runner_shell_test.rb > output.txt
$ cat output.txt
Hello, world! 2 times!
Hello, world! 2 times!
Hello, world! 2 times!

It should only print two times...

I don't think you said anything about the platform you are running under.
When I run your test under ruby 1.8.2p2 and FreeBSD 4.10, I get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

and uncommenting the section labelled "more wackiness", I get

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
Hello, world! 2 times!
Hello, world! 3 times!
Hello, world! 3 times!
Hello, world! 3 times!
$

which looks pretty good to me too.

If you are running under cygwin plus some Microsoft operating system, then
there is a quotation provided to me by the 'fortune' program which I think
is relevant:

--------------------------------------------------------------------------
There's a long-standing bug relating to the x86 architecture that
allows you to install Windows.
-- Matthew D. Fuller
--------------------------------------------------------------------------

:)

Seriously, Windows is badly broken with regard to processes and pipes.
All the weird thread and waiting for the 'R' character, trickery is so
that I can guaranty that by the time I am calling the waitpid function,
the process and PID still exist. In other words, it is possible for the
child process to be so quick that it finishes executing before waitpid()
gets called). So on a sidenote, does anyone know a cleaner way of
performing interprocess communication? Like how do I tell the forked,
child process that the parent is ready and waiting.

Why is this a problem? You can call waitpid on the child pid, and if it has
already terminated, then it will still be in the process table (as a zombie)
waiting for you to reap its exit status. That's unless someone has messed
with SIGCHLD.

You can prove this for yourself with a simple test program:

child = fork do
exit 42 # die immediately
end

sleep 5

Process.waitpid(child)
p $?

When I run this under FreeBSD, I get a delay of 5 seconds followed by
#<Process::Status: pid=1365,exited(42)>

If you use ps to see what is happening during the sleep, what I see is:

brian 1386 0.0 0.4 2544 1956 p7 S 11:15AM 0:00.01 ruby forktest.rb
brian 0 0.0 0.0 0 0 p7 ZW - 0:00.00 (ruby)
^
the zombie, waiting for collection

And even if the process had gone completely, you'd just get an error from
waitpid saying that the child did not exist (which is fine, since you know
it did exist at the time you forked, so it must have died since then). Only
if another process on the system had forked and re-used the same PID in the
mean time would that be a problem. Sensible Unixes are fine for this because
they allocate PIDs sequentially, and it's likely to be several minutes
before the same PID is re-used; however OpenBSD is completely broken, and
*could* re-use a PID within a microsecond of the previous child dying.

Regards,

Brian.
 
B

Brian Candler

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

Oops, copy-paste error. I actually get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
Hello, world! 2 times!
$
 
S

Steven Kah Hien Wong

----=_NextPart_ST_10_14_03_Tuesday_November_02_2004_23267
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi Brian,

Firstly, thanks for the help.
I don't think you said anything about the platform you are running under.
When I run your test under ruby 1.8.2p2 and FreeBSD 4.10, I get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

which looks pretty good to me too.

You actually need to run the test script with a shell redirection. Like
in the example output I gave, I redirected output of the script to a
file, output.txt, and then cat'ed it:

$ ruby command_runner_shell_test.rb > output.txt
$ cat output.txt
Hello, world! 2 times!
Hello, world! 2 times!
Hello, world! 2 times!

Hopefully that will reproduce the same results I am seeing? Otherwise if
this problem just happens on mine... well that is just going to make it
even more difficult for me to track. :(
If you are running under cygwin plus some Microsoft operating system

I am running this on Debian Linux 3.1, the testing branch of the
distribution. Ruby v1.8.2. Maybe the Debian package maintainers applied
some patches, but I doubt that would be the cause... but maybe.
Seriously, Windows is badly broken with regard to processes and pipes.

I would agree there. I once had to write a sh-like shell that compiled
and ran on both Windows and Linux, for a uni project. :) I never
finished the windows part by the deadline (I did start a bit last minute
though)...
Why is this a problem? You can call waitpid on the child pid, and if it has
already terminated, then it will still be in the process table (as a zombie)
waiting for you to reap its exit status. That's unless someone has messed
with SIGCHLD.

That is just the handy little fact I needed. Now I guess I can remove
all that part of the code and be confident it works. Thanks!
And even if the process had gone completely, you'd just get an error from
waitpid saying that the child did not exist (which is fine, since you know
it did exist at the time you forked, so it must have died since then). Only
if another process on the system had forked and re-used the same PID in the
mean time would that be a problem.

Which was the (unlikely) problem I was trying to protect myself from. I
know it won't happen... but the fact that it "technically" could made me
uncomfortable. :p

I've attached the 'updated' command_runner with the threading stuff
removed. Hopefully with that cruft gone, it is easier to read.

Steven
--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E (e-mail address removed)
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.
----=_NextPart_ST_10_14_03_Tuesday_November_02_2004_23267
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="command_runner.rb"

class CommandRunner
attr :command
attr :childPid

def initialize(*command)
@command = command
@childPid = nil

@readPipe = nil
@readErrorPipe = nil
@writePipe = nil
end

def closeWrite
@writePipe.close
end

def kill
Process.kill("KILL", @childPid)
end

def read
return @readPipe.read
end

def readError
return @readErrorPipe.read
end

def run
parent_to_child_read, parent_to_child_write = IO.pipe
child_to_parent_read, child_to_parent_write = IO.pipe
child_to_parent_error_read, child_to_parent_error_write = IO.pipe

@childPid = fork do
parent_to_child_write.close
child_to_parent_read.close
child_to_parent_error_read.close

$stdin.reopen(parent_to_child_read) or
raise "Unable to redirect STDIN"
$stdout.reopen(child_to_parent_write) or
raise "Unable to redirect STDOUT"
$stderr.reopen(child_to_parent_error_write) or
raise "Unable to redirect STDERR"

exec(*@command)
end

child_to_parent_write.close
child_to_parent_error_write.close
parent_to_child_read.close

@readPipe = child_to_parent_read
@readErrorPipe = child_to_parent_error_read
@writePipe = parent_to_child_write
end

#--------------------------------------------------------------------------
# Description: Waits for command to exit
# Returns : The return code of the program when it exited
#--------------------------------------------------------------------------
def wait
if not @childPid
raise "Waiting for a process that has not started"
end

return_value = Process.waitpid2(@childPid)[1].exitstatus

@childPid = nil

return return_value
end

def write(string)
@writePipe.write(string)
end
end

----=_NextPart_ST_10_14_03_Tuesday_November_02_2004_23267--
 
Y

Yukihiro Matsumoto

Hi,

In message "Re: Ruby redirection woes"

|You actually need to run the test script with a shell redirection. Like
|in the example output I gave, I redirected output of the script to a
|file, output.txt, and then cat'ed it:
|
| $ ruby command_runner_shell_test.rb > output.txt
| $ cat output.txt
| Hello, world! 2 times!
| Hello, world! 2 times!
| Hello, world! 2 times!
|
|Hopefully that will reproduce the same results I am seeing? Otherwise if
|this problem just happens on mine... well that is just going to make it
|even more difficult for me to track. :(

Both parent and child processes flushes remaining output buffer. We
have to flush stdout and stderr before fork(2). Here's the patch.

matz.
--- process.c 2 Oct 2004 03:50:48 -0000 1.92.2.11
+++ process.c 2 Nov 2004 07:07:49 -0000
@@ -1259,2 +1259,8 @@ rb_f_fork(obj)
rb_secure(2);
+
+#ifndef __VMS
+ fflush(stdout);
+ fflush(stderr);
+#endif
+
switch (pid = fork()) {
 
S

Steven Kah Hien Wong

Both parent and child processes flushes remaining output buffer. We
have to flush stdout and stderr before fork(2). Here's the patch.

Hi Matz,

That was fast. Just tested it on my box with Ruby-1.8.2-preview2, and it
seems to work:

$ ~/local/bin/ruby command_runner_shell_test.rb > out.txt
$ cat out.txt
Hello, world! 2 times!
Hello, world! 2 times!

All good. :)

That's one less thing to scratch off my agenda of things to do.

Thank you! :)

Steven

P.S. I am guessing this will be merged into the next release?

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E (e-mail address removed)
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.
 
Y

Yukihiro Matsumoto

Hi,

In message "Re: Ruby redirection woes"

|P.S. I am guessing this will be merged into the next release?

Yes. It already checked in to the stable CVS.

matz.
 
B

Brian Candler

You actually need to run the test script with a shell redirection. Like
in the example output I gave, I redirected output of the script to a
file, output.txt, and then cat'ed it:

$ ruby command_runner_shell_test.rb > output.txt
$ cat output.txt

My apologies for not reading your post more carefully. Yes, I get the same
here - and Matz has posted a patch which fixes it in Ruby. (However I'm not
sure that fix is necessarily good; the same problem would happen with other
filehandles which are open, and there's no proposal that fork() should flush
all those too).

However you can also just fix your command_runner.rb like this:

--- command_runner.rb.orig Tue Nov 2 13:21:01 2004
+++ command_runner.rb Tue Nov 2 13:21:41 2004
@@ -33,6 +33,8 @@
child_to_parent_read, child_to_parent_write = IO.pipe
child_to_parent_error_read, child_to_parent_error_write = IO.pipe

+ $stdout.flush
+ $stderr.flush
@childPid = fork do
parent_to_child_write.close
child_to_parent_read.close

(The $stderr.flush maybe isn't needed, as I seem to remember that stderr
is unbuffered normally, but it won't do any harm to have it there)

Regards,

Brian.
 
S

Steven Kah Hien Wong

(However I'm not sure that fix is necessarily good; the same problem
would happen with other filehandles which are open, and there's no
proposal that fork() should flush all those too).

Yes, I guess you have a point there. I guess fork should be left as it
was before, then? Perhaps add documentation to flush stdout before
forking in the Ruby documentation somewhere.

Never knew forking could get so complicated. :(

Steven
--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E (e-mail address removed)
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top