Asynchronous process to process pipe IO

Discussion in 'Ruby' started by Pito Salas, Aug 24, 2009.

  1. Pito Salas

    Pito Salas Guest

    This is slightly subtle question about how a process that reads from a
    pipe that is being written to by another process may run asynchronously.

    Here's a code snippet:

    def start_processing
    open("|-", "r") do |worker|
    if worker
    # here we are in the parent
    i = 0
    worker.each_line do |line|
    puts "line #{i}
    i = i+1
    end
    else
    # here we are in child thread
    exec("./iacommand.rb")
    end
    end
    end

    iacommand.rb is invoked as a process within ruby. It does a lot of
    processing, outputs a line, does a more processing, outputs another
    line, and so on.

    I would like to have the "puts" calls occur right when the a line is
    generated by the script running in the process.

    The way it is written above, what happens is that the app waits until
    iacommand.rb exits and then calls the series of puts in immediate
    succession.

    So in other words, how do I asynchronously read from a pipe that is
    being fed by a process?
    --
    Posted via http://www.ruby-forum.com/.
     
    Pito Salas, Aug 24, 2009
    #1
    1. Advertising

  2. Pito Salas wrote:
    > This is slightly subtle question about how a process that reads from a
    > pipe that is being written to by another process may run asynchronously.
    >
    > Here's a code snippet:
    >
    > def start_processing
    > open("|-", "r") do |worker|
    > if worker
    > # here we are in the parent
    > i = 0
    > worker.each_line do |line|
    > puts "line #{i}
    > i = i+1
    > end
    > else
    > # here we are in child thread
    > exec("./iacommand.rb")
    > end
    > end
    > end
    >
    > iacommand.rb is invoked as a process within ruby. It does a lot of
    > processing, outputs a line, does a more processing, outputs another
    > line, and so on.
    >
    > I would like to have the "puts" calls occur right when the a line is
    > generated by the script running in the process.
    >
    > The way it is written above, what happens is that the app waits until
    > iacommand.rb exits and then calls the series of puts in immediate
    > succession.
    >
    > So in other words, how do I asynchronously read from a pipe that is
    > being fed by a process?


    Hm, the following shows output at 1 sec intervals:

    open("|-", "r") do |worker|
    if worker
    i = 0
    worker.each_line do |line|
    puts "line #{i}:#{line}"
    i = i+1
    end
    else
    exec("ruby -e '3.times {|i| p i; sleep 1}'")
    end
    end

    Does it work that way for you?

    --
    vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407
     
    Joel VanderWerf, Aug 24, 2009
    #2
    1. Advertising

  3. Pito Salas

    Pito Salas Guest

    Joel VanderWerf wrote:
    > Pito Salas wrote:
    >> worker.each_line do |line|
    >> iacommand.rb is invoked as a process within ruby. It does a lot of
    >> So in other words, how do I asynchronously read from a pipe that is
    >> being fed by a process?

    >
    > Hm, the following shows output at 1 sec intervals:
    >
    > open("|-", "r") do |worker|
    > if worker
    > i = 0
    > worker.each_line do |line|
    > puts "line #{i}:#{line}"
    > i = i+1
    > end
    > else
    > exec("ruby -e '3.times {|i| p i; sleep 1}'")
    > end
    > end
    >
    > Does it work that way for you?


    Yes it does. Double-hm.

    My case is using a second ruby file, iacommand.rb, which is doing much
    the same as what your -e is doing. I wonder if that makes the
    difference? Or it must be something else.
    --
    Posted via http://www.ruby-forum.com/.
     
    Pito Salas, Aug 24, 2009
    #3
  4. Pito Salas

    Pito Salas Guest

    Pito Salas wrote:

    > Yes it does. Double-hm.
    >
    > My case is using a second ruby file, iacommand.rb, which is doing much
    > the same as what your -e is doing. I wonder if that makes the
    > difference? Or it must be something else.


    It does seem to be the difference between ruby -e and ./iacommand.rb.
    Can anyone see what the key is??

    Thanks!!

    Here's what fails (that is, blocks) (tested both in irb and in eclipse)

    open("|-", "r") do |worker|
    if worker
    i = 0
    worker.each_line do |line|
    puts "line #{i}:#{line}"
    i = i+1
    end
    else
    exec("./iacommand.rb", "-t")
    end
    end


    And here's the exact text of iacommand.rb:


    require 'rubygems'
    require 'getoptlong'

    parser = GetoptLong.new
    parser.set_options(
    ["-h", "--help", GetoptLong::NO_ARGUMENT],
    ["-t", "--test", GetoptLong::NO_ARGUMENT],
    ["-v", "--version", GetoptLong::NO_ARGUMENT])

    valid = false
    loop do

    begin
    opt, arg = parser.get
    break if not opt
    case opt
    when "-h"
    puts "Usage: ..."
    valid = true
    break
    when "-t"
    valid = true
    break
    when "-v"
    puts "Version 0.0"
    valid = true
    break
    end
    end
    end

    if valid
    puts "start"
    i = 0
    5.times do
    puts "ballot #{i}"
    delay = rand(5).to_i
    sleep delay
    i = i+1
    end
    puts "exit"
    else
    puts "invalid parameters for iacommand"
    end




    --
    Posted via http://www.ruby-forum.com/.
     
    Pito Salas, Aug 24, 2009
    #4
  5. On 24.08.2009 22:16, Joel VanderWerf wrote:
    > Pito Salas wrote:
    >> This is slightly subtle question about how a process that reads from a
    >> pipe that is being written to by another process may run asynchronously.
    >>
    >> Here's a code snippet:
    >>
    >> def start_processing
    >> open("|-", "r") do |worker|
    >> if worker
    >> # here we are in the parent
    >> i = 0
    >> worker.each_line do |line|
    >> puts "line #{i}
    >> i = i+1
    >> end
    >> else
    >> # here we are in child thread
    >> exec("./iacommand.rb")
    >> end
    >> end
    >> end
    >>
    >> iacommand.rb is invoked as a process within ruby. It does a lot of
    >> processing, outputs a line, does a more processing, outputs another
    >> line, and so on.
    >>
    >> I would like to have the "puts" calls occur right when the a line is
    >> generated by the script running in the process.
    >>
    >> The way it is written above, what happens is that the app waits until
    >> iacommand.rb exits and then calls the series of puts in immediate
    >> succession.
    >>
    >> So in other words, how do I asynchronously read from a pipe that is
    >> being fed by a process?

    >
    > Hm, the following shows output at 1 sec intervals:
    >
    > open("|-", "r") do |worker|
    > if worker
    > i = 0
    > worker.each_line do |line|
    > puts "line #{i}:#{line}"
    > i = i+1
    > end
    > else
    > exec("ruby -e '3.times {|i| p i; sleep 1}'")
    > end
    > end
    >
    > Does it work that way for you?


    Why are you guys using such a complex construction? If you just want to
    see the output when it comes why not just:

    system "./iacommand.rb"

    Or maybe this, if you need to process the output:

    IO.popen "./iacommand.rb" do |io|
    io.each_line do |line|
    puts line
    end
    end

    Note, you may have to do $stdout.sync = true in "iacommand.rb".

    Kind regards

    robert

    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
     
    Robert Klemme, Aug 24, 2009
    #5
  6. Pito Salas

    Pito Salas Guest

    Robert Klemme wrote:
    > On 24.08.2009 22:16, Joel VanderWerf wrote:
    >>> i = 0
    >>>
    >>>

    >> i = i+1
    >> end
    >> else
    >> exec("ruby -e '3.times {|i| p i; sleep 1}'")
    >> end
    >> end
    >>
    >> Does it work that way for you?

    >
    > Why are you guys using such a complex construction? If you just want to
    > see the output when it comes why not just:
    >
    > system "./iacommand.rb"
    >
    > Or maybe this, if you need to process the output:
    >
    > IO.popen "./iacommand.rb" do |io|
    > io.each_line do |line|
    > puts line
    > end
    > end
    >
    > Note, you may have to do $stdout.sync = true in "iacommand.rb".
    >
    > Kind regards
    >
    > robert


    Robert,

    Thanks, the $stdout.sync = true was the magic bullet.

    Now, I am not exactly clear (reading the doc) about the difference
    between exec and system... I do need to be able to send command line
    arguments to iacommand.rb and I do indeed need to process the output 'as
    it appears.'

    Thanks!!

    Pito

    --
    Posted via http://www.ruby-forum.com/.
     
    Pito Salas, Aug 24, 2009
    #6
  7. Pito Salas

    Gary Wright Guest

    On Aug 24, 2009, at 4:54 PM, Pito Salas wrote:
    >
    > Now, I am not exactly clear (reading the doc) about the difference
    > between exec and system... I do need to be able to send command line
    > arguments to iacommand.rb and I do indeed need to process the output
    > 'as
    > it appears.'


    exec() causes a new program to be executed within the current process.
    Unless exec() fails (e.g. program not found), exec() will never 'return'
    because the current program will be discarded (i.e. the Ruby
    interpreter)
    in favor of the newly exec'ed one.

    system() causes a new (child) process to be created and for the new
    program to be executed (i.e. exec'ed) within the new child process.
    The current process will wait for the child process to terminate before
    proceeding.

    The difference is whether the new program replaces the currently
    running program or not. Note that 'program' here means the Ruby
    interpreter and not the particular Ruby script that is running.

    Note: I'm answering from the perspective of a Unix/Posix environment.
    My windows-fu isn't sufficient to say if my description is correct
    for that environment, but it is probably pretty close.

    Gary Wright
     
    Gary Wright, Aug 24, 2009
    #7
  8. On Aug 24, 1:05 pm, Gary Wright <> wrote:
    > On Aug 24, 2009, at 4:54 PM, Pito Salas wrote:
    >
    >
    >
    > > Now, I am not exactly clear (reading the doc) about the difference
    > > between exec and system... I do need to be able to send command line
    > > arguments to iacommand.rb and I do indeed need to process the output  
    > > 'as
    > > it appears.'

    >
    > exec() causes a new program to be executed within the current process.
    > Unless exec() fails (e.g. program not found), exec() will never 'return'
    > because the current program will be discarded (i.e. the Ruby  
    > interpreter)
    > in favor of the newly exec'ed one.
    >
    > system() causes a new (child) process to be created and for the new
    > program to be executed (i.e. exec'ed) within the new child process.
    > The current process will wait for the child process to terminate before
    > proceeding.
    >
    > The difference is whether the new program replaces the currently
    > running program or not. Note that 'program' here means the Ruby
    > interpreter and not the particular Ruby script that is running.
    >
    > Note: I'm answering from the perspective of a Unix/Posix environment.
    > My windows-fu isn't sufficient to say if my description is correct
    > for that environment, but it is probably pretty close.
    >
    > Gary Wright


    Would Ruby's fork behave exactly like system here, or would it have a
    third path? Likely fork wouldn't be an option for the OP because he
    wants to process the output?

    Mike B.
     
    Mike Barsalou, Aug 24, 2009
    #8
  9. Pito Salas

    Gary Wright Guest

    On Aug 24, 2009, at 6:50 PM, Mike Barsalou wrote:
    >
    > Would Ruby's fork behave exactly like system here, or would it have a
    > third path? Likely fork wouldn't be an option for the OP because he
    > wants to process the output?


    fork and exec are the 'primitive' operations.

    system is basically implemented by forking and having the child
    process exec the new program while the parent waits for it to
    complete. System() is just a convenience method for this
    common pattern.

    Capturing the output of a program basically means using fork/exec
    along with rearranging the standout output of the exec'ed
    program. For simple cases this is handled by Ruby's backtick syntax:

    output = `program_to_run`

    Gary Wright
     
    Gary Wright, Aug 25, 2009
    #9
  10. On 24.08.2009 22:54, Pito Salas wrote:
    > Robert Klemme wrote:
    >> On 24.08.2009 22:16, Joel VanderWerf wrote:
    >>>> i = 0
    >>>>
    >>>>
    >>> i = i+1
    >>> end
    >>> else
    >>> exec("ruby -e '3.times {|i| p i; sleep 1}'")
    >>> end
    >>> end
    >>>
    >>> Does it work that way for you?

    >> Why are you guys using such a complex construction? If you just want to
    >> see the output when it comes why not just:
    >>
    >> system "./iacommand.rb"
    >>
    >> Or maybe this, if you need to process the output:
    >>
    >> IO.popen "./iacommand.rb" do |io|
    >> io.each_line do |line|
    >> puts line
    >> end
    >> end
    >>
    >> Note, you may have to do $stdout.sync = true in "iacommand.rb".
    >>
    >> Kind regards
    >>
    >> robert

    >
    > Robert,
    >
    > Thanks, the $stdout.sync = true was the magic bullet.
    >
    > Now, I am not exactly clear (reading the doc) about the difference
    > between exec and system... I do need to be able to send command line
    > arguments to iacommand.rb and I do indeed need to process the output 'as
    > it appears.'


    Then you'll want to use one of the popen family of functions. Even with
    IO.popen you can pass arguments:

    IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
    ...
    end

    Note, if you also want to write to the process's stdin, you need to
    provide a file mode as second argument. See docs.

    Kind regards

    robert

    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
     
    Robert Klemme, Aug 25, 2009
    #10
  11. Pito Salas

    Pito Salas Guest

    Robert Klemme wrote:

    > Then you'll want to use one of the popen family of functions. Even with
    > IO.popen you can pass arguments:
    >
    > IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
    > ...
    > end
    >
    > Note, if you also want to write to the process's stdin, you need to
    > provide a file mode as second argument. See docs.


    This is what I originally had. Pretty similar...

    open("|-", "r") do |worker|
    if worker
    ...
    else
    exec("./iacommand.rb", "arg", "arg")
    end
    end

    How do you think that the idea you show above is different/better? This
    gets pretty subtle... Thanks again for the great thread about Threads :)
    --
    Posted via http://www.ruby-forum.com/.
     
    Pito Salas, Aug 25, 2009
    #11
  12. 2009/8/25 Pito Salas <>:
    > Robert Klemme wrote:
    >
    >> Then you'll want to use one of the popen family of functions. =A0Even wi=

    th
    >> IO.popen you can pass arguments:
    >>
    >> IO.popen ["./iacommand.rb", "arg", "arg"] do |io|
    >> =A0 =A0...
    >> end
    >>
    >> Note, if you also want to write to the process's stdin, you need to
    >> provide a file mode as second argument. =A0See docs.

    >
    > This is what I originally had. Pretty similar...
    >
    > open("|-", "r") do |worker|
    > =A0 =A0if worker
    > ...
    > =A0 =A0else
    > =A0 =A0 =A0exec("./iacommand.rb", "arg", "arg")
    > =A0 =A0end
    > =A0end
    >
    > How do you think that the idea you show above is different/better?


    I would prefer IO.popen as it is shorter - and it does not need
    cryptic (aka perlish) file names:

    IO.popen ["./iacommand.rb", "arg", "arg"], "rw" do |io|
    # no if here either
    io.puts "start"
    io.close_write

    io.each_line do |line|
    puts "GOT: #{line}"
    end
    end

    > This
    > gets pretty subtle... Thanks again for the great thread about Threads :)


    Sorry, I do not find it that subtle. IMHO it is pretty obvious that
    IO.popen is superfior for the job you want to get done. That is, of
    course, unless I am missing something...

    Kind regards

    robert

    --=20
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
     
    Robert Klemme, Aug 26, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jacob
    Replies:
    7
    Views:
    12,664
    Roedy Green
    Apr 30, 2004
  2. Hoegje
    Replies:
    2
    Views:
    22,597
    Gianni Mariani
    Dec 5, 2003
  3. lee, wonsun
    Replies:
    1
    Views:
    503
    Jack Klein
    Nov 2, 2004
  4. Jill Graham

    connection.execute = asynchronous process ??

    Jill Graham, Oct 3, 2003, in forum: ASP General
    Replies:
    4
    Views:
    214
    Ray at
    Oct 4, 2003
  5. Replies:
    1
    Views:
    246
    Ben Morrow
    Jun 2, 2004
Loading...

Share This Page