working with a slow pipe (IO.popen)

Discussion in 'Ruby' started by bwv549, Dec 16, 2009.

  1. bwv549

    bwv549 Guest

    I'm doing some work with pymol and having trouble getting all the
    pymol output out:

    # I open it in quiet/commandline mode with a pipe:
    IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    # this command will generate a whole bunch of output to stdout
    pipe.puts "my_script mymodel"
    sleep(5) # <--- this is what I have to do in order to get the
    output
    pipe.close_write
    output = pipe.read
    end

    What is annoying is that unless I sleep for 2-5 seconds I don't get
    output or (even worse) my output is cut off. There has to be a way
    besides arbitrarily sleeping to ensure that the command is finished.
    I've tried things like piping in "quit", but that doesn't seem to
    work. Anybody know how to solve this? I'm happy to use a different
    approach if it means I can be sure I get all of my output out. --
    Thanks!
    bwv549, Dec 16, 2009
    #1
    1. Advertising

  2. bwv549

    Eric Wong Guest

    bwv549 <> wrote:
    > I'm doing some work with pymol and having trouble getting all the
    > pymol output out:
    >
    > # I open it in quiet/commandline mode with a pipe:
    > IO.popen("pymol -cq -p", 'w+') do |pipe|
    > pipe.puts "load file.pdb, mymodel\n"
    > pipe.puts "run my_script.py"
    > # this command will generate a whole bunch of output to stdout
    > pipe.puts "my_script mymodel"
    > sleep(5) # <--- this is what I have to do in order to get the
    > output
    > pipe.close_write
    > output = pipe.read
    > end
    >
    > What is annoying is that unless I sleep for 2-5 seconds I don't get
    > output or (even worse) my output is cut off. There has to be a way
    > besides arbitrarily sleeping to ensure that the command is finished.
    > I've tried things like piping in "quit", but that doesn't seem to
    > work. Anybody know how to solve this? I'm happy to use a different
    > approach if it means I can be sure I get all of my output out. --
    > Thanks!


    Hi,

    I don't know about pymol, but some apps do not respond well to having
    its stdin closed on it with close_write. pipe.read should just block
    until you get output, otherwise put an IO.select([pipe) in front of it.

    Does this work?

    IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    pipe.puts "my_script mymodel"

    # IO.select([pipe]) # you shouldn't need to uncomment this...

    output = pipe.read
    end

    --
    Eric Wong
    Eric Wong, Dec 16, 2009
    #2
    1. Advertising

  3. bwv549

    bwv549 Guest

    On Dec 15, 11:32 pm, Eric Wong <> wrote:
    > bwv549 <> wrote:
    > > I'm doing some work with pymol and having trouble getting all the
    > > pymol output out:

    >
    > > # I open it in quiet/commandline mode with a pipe:
    > > IO.popen("pymol -cq -p", 'w+') do |pipe|
    > >   pipe.puts "load file.pdb, mymodel\n"
    > >   pipe.puts "run my_script.py"
    > >   # this command will generate a whole bunch of output to stdout
    > >   pipe.puts "my_script mymodel"
    > >   sleep(5)  # <--- this is what I have to do in order to get the
    > > output
    > >   pipe.close_write
    > >   output = pipe.read
    > > end

    >
    > > What is annoying is that unless I sleep for 2-5 seconds I don't get
    > > output or (even worse) my output is cut off.  There has to be a way
    > > besides arbitrarily sleeping to ensure that the command is finished.
    > > I've tried things like piping in "quit", but that doesn't seem to
    > > work.  Anybody know how to solve this?  I'm happy to use a different
    > > approach if it means I can be sure I get all of my output out.   --
    > > Thanks!

    >
    > Hi,
    >
    > I don't know about pymol, but some apps do not respond well to having
    > its stdin closed on it with close_write.  pipe.read should just block
    > until you get output, otherwise put an IO.select([pipe) in front of it.
    >
    > Does this work?
    >
    >   IO.popen("pymol -cq -p", 'w+') do |pipe|
    >     pipe.puts "load file.pdb, mymodel\n"
    >     pipe.puts "run my_script.py"
    >     pipe.puts "my_script mymodel"
    >
    >     # IO.select([pipe]) # you shouldn't need to uncomment this...
    >
    >     output = pipe.read
    >   end
    >
    > --
    > Eric Wong


    both of those hang indefinitely. It doesn't hang if I add the line:

    pipe.puts "quit"

    but that doesn't give me all the output either. So maybe pymol
    behaves differently than many commandline programs?

    Here is what I've worked out for the time being. It opens a thread
    for reading the output and asks if there is any output every 1/2
    second. Once we go 1/2 second without output we kill the thread.

    my_string = ""
    Open3.popen3("pymol -cq -p") do |si, so, se|
    si.puts "load file.pdb, mymodel\n"
    si.puts "run my_script.py\n"
    si.puts "myscript mymodel\n"

    forstdout = Thread.new do
    Thread.current['lines'] = []
    while line = so.gets
    Thread.current['lines'] << line
    end
    end

    past_size = -1
    loop do
    sleep(0.5)
    current_size = forstdout['lines'].size
    break if current_size == past_size
    past_size = current_size
    end
    my_string = forstdout['lines']
    forstdout.kill
    end

    This is not the fastest method, but it does seem to work OK. I would
    still love to hear other ideas.
    bwv549, Dec 16, 2009
    #3
  4. bwv549

    Eric Wong Guest

    bwv549 <> wrote:
    > On Dec 15, 11:32 pm, Eric Wong <> wrote:
    > > bwv549 <> wrote:
    > > > I'm doing some work with pymol and having trouble getting all the
    > > > pymol output out:

    > >
    > > > # I open it in quiet/commandline mode with a pipe:
    > > > IO.popen("pymol -cq -p", 'w+') do |pipe|
    > > >   pipe.puts "load file.pdb, mymodel\n"
    > > >   pipe.puts "run my_script.py"
    > > >   # this command will generate a whole bunch of output to stdout
    > > >   pipe.puts "my_script mymodel"
    > > >   sleep(5)  # <--- this is what I have to do in order to get the
    > > > output
    > > >   pipe.close_write
    > > >   output = pipe.read
    > > > end

    > >
    > > > What is annoying is that unless I sleep for 2-5 seconds I don't get
    > > > output or (even worse) my output is cut off.  There has to be a way
    > > > besides arbitrarily sleeping to ensure that the command is finished.
    > > > I've tried things like piping in "quit", but that doesn't seem to
    > > > work.  Anybody know how to solve this?  I'm happy to use a different
    > > > approach if it means I can be sure I get all of my output out.   --
    > > > Thanks!

    > >
    > > Hi,
    > >
    > > I don't know about pymol, but some apps do not respond well to having
    > > its stdin closed on it with close_write.  pipe.read should just block
    > > until you get output, otherwise put an IO.select([pipe) in front of it.
    > >
    > > Does this work?
    > >
    > >   IO.popen("pymol -cq -p", 'w+') do |pipe|
    > >     pipe.puts "load file.pdb, mymodel\n"
    > >     pipe.puts "run my_script.py"
    > >     pipe.puts "my_script mymodel"
    > >
    > >     # IO.select([pipe]) # you shouldn't need to uncomment this...
    > >
    > >     output = pipe.read
    > >   end

    >
    > both of those hang indefinitely. It doesn't hang if I add the line:
    >
    > pipe.puts "quit"
    >
    > but that doesn't give me all the output either. So maybe pymol
    > behaves differently than many commandline programs?


    Even IO.select([pipe]) hangs indefinitely? Oh, I guess pymol
    expects an explicit quit of some sort.... What if you did:

    IO.select([pipe])
    pipe.readpartial(whatever_size_you_are_comfortable_with)

    > Here is what I've worked out for the time being. It opens a thread
    > for reading the output and asks if there is any output every 1/2
    > second. Once we go 1/2 second without output we kill the thread.


    Here's a version without threads, should work the same:

    my_string = ""
    Open3.popen3("pymol -cq -p") do |si, so, se|
    si.puts "load file.pdb, mymodel\n"
    si.puts "run my_script.py\n"
    si.puts "myscript mymodel\n"

    # await input for 0.5 seconds, will return nil and
    # break the loop if there is nothing to read from so after 0.5s
    while ready = IO.select([so], nil, nil, 0.5)
    # ready.first == so # in this case

    # read until the current pipe buffer is empty
    begin
    my_string << so.read_nonblock(4096)
    rescue Errno::EAGAIN
    break
    end while true
    end
    end

    --
    Eric Wong
    Eric Wong, Dec 16, 2009
    #4
  5. 2009/12/16 bwv549 <>:
    > On Dec 15, 11:32=A0pm, Eric Wong <> wrote:
    >> bwv549 <> wrote:
    >> > I'm doing some work with pymol and having trouble getting all the
    >> > pymol output out:

    >>
    >> > # I open it in quiet/commandline mode with a pipe:
    >> > IO.popen("pymol -cq -p", 'w+') do |pipe|
    >> > =A0 pipe.puts "load file.pdb, mymodel\n"
    >> > =A0 pipe.puts "run my_script.py"
    >> > =A0 # this command will generate a whole bunch of output to stdout
    >> > =A0 pipe.puts "my_script mymodel"
    >> > =A0 sleep(5) =A0# <--- this is what I have to do in order to get the
    >> > output
    >> > =A0 pipe.close_write
    >> > =A0 output =3D pipe.read
    >> > end

    >>
    >> > What is annoying is that unless I sleep for 2-5 seconds I don't get
    >> > output or (even worse) my output is cut off. =A0There has to be a way
    >> > besides arbitrarily sleeping to ensure that the command is finished.
    >> > I've tried things like piping in "quit", but that doesn't seem to
    >> > work. =A0Anybody know how to solve this? =A0I'm happy to use a differe=

    nt
    >> > approach if it means I can be sure I get all of my output out. =A0 --
    >> > Thanks!

    >>
    >> Hi,
    >>
    >> I don't know about pymol, but some apps do not respond well to having
    >> its stdin closed on it with close_write. =A0pipe.read should just block
    >> until you get output, otherwise put an IO.select([pipe) in front of it.
    >>
    >> Does this work?
    >>
    >> =A0 IO.popen("pymol -cq -p", 'w+') do |pipe|
    >> =A0 =A0 pipe.puts "load file.pdb, mymodel\n"
    >> =A0 =A0 pipe.puts "run my_script.py"
    >> =A0 =A0 pipe.puts "my_script mymodel"
    >>
    >> =A0 =A0 # IO.select([pipe]) # you shouldn't need to uncomment this...
    >>
    >> =A0 =A0 output =3D pipe.read
    >> =A0 end
    >>
    >> --
    >> Eric Wong

    >
    > both of those hang indefinitely. =A0It doesn't hang if I add the line:
    >
    > =A0 pipe.puts "quit"
    >
    > but that doesn't give me all the output either. =A0So maybe pymol
    > behaves differently than many commandline programs?


    What does the documentation of "pymol" say?

    > Here is what I've worked out for the time being. =A0It opens a thread
    > for reading the output and asks if there is any output every 1/2
    > second. =A0Once we go 1/2 second without output we kill the thread.
    >
    > my_string =3D ""
    > Open3.popen3("pymol -cq -p") do |si, so, se|
    > =A0si.puts "load file.pdb, mymodel\n"
    > =A0si.puts "run my_script.py\n"
    > =A0si.puts "myscript mymodel\n"
    >
    > =A0forstdout =3D Thread.new do
    > =A0 =A0Thread.current['lines'] =3D []
    > =A0 =A0while line =3D so.gets
    > =A0 =A0 =A0Thread.current['lines'] << line
    > =A0 =A0end
    > =A0end


    You don't need the thread local for this to work. Note there is also
    Thread#value!

    How does this work:

    output =3D IO.popen("pymol -cq -p", 'w+') do |pipe|
    reader =3D Thread.new { pipe.to_a }

    pipe.puts "load file.pdb, mymodel"
    pipe.puts "run my_script.py"
    # this command will generate a whole bunch of output to stdout
    pipe.puts "my_script mymodel"

    pipe.close_write
    reader.value
    end

    Kind regards

    robert

    --=20
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Dec 16, 2009
    #5
  6. bwv549

    bwv549 Guest

    @Robert: I've looked through a lot of pymol docs and haven't found
    anything on it yet, will post if I do.

    The solution you post looks very intriguing. It runs fine, but closes
    before I get back the output I need. It seems like there must be some
    kind of a pause to let pymol start spitting out output. I may play
    around with it some to see if I can get it working.

    @Eric: this solution works nicely and returns all the expected
    output. I will be using it unless something better comes along.

    Thanks to both of you for your excellent input on this. --John


    On Dec 16, 1:05 am, Robert Klemme <> wrote:
    > 2009/12/16 bwv549 <>:
    >
    >
    >
    > > On Dec 15, 11:32 pm, Eric Wong <> wrote:
    > >> bwv549 <> wrote:
    > >> > I'm doing some work with pymol and having trouble getting all the
    > >> > pymol output out:

    >
    > >> > # I open it in quiet/commandline mode with a pipe:
    > >> > IO.popen("pymol -cq -p", 'w+') do |pipe|
    > >> >   pipe.puts "load file.pdb, mymodel\n"
    > >> >   pipe.puts "run my_script.py"
    > >> >   # this command will generate a whole bunch of output to stdout
    > >> >   pipe.puts "my_script mymodel"
    > >> >   sleep(5)  # <--- this is what I have to do in order to get the
    > >> > output
    > >> >   pipe.close_write
    > >> >   output = pipe.read
    > >> > end

    >
    > >> > What is annoying is that unless I sleep for 2-5 seconds I don't get
    > >> > output or (even worse) my output is cut off.  There has to be a way
    > >> > besides arbitrarily sleeping to ensure that the command is finished.
    > >> > I've tried things like piping in "quit", but that doesn't seem to
    > >> > work.  Anybody know how to solve this?  I'm happy to use a different
    > >> > approach if it means I can be sure I get all of my output out.   --
    > >> > Thanks!

    >
    > >> Hi,

    >
    > >> I don't know about pymol, but some apps do not respond well to having
    > >> its stdin closed on it with close_write.  pipe.read should just block
    > >> until you get output, otherwise put an IO.select([pipe) in front of it..

    >
    > >> Does this work?

    >
    > >>   IO.popen("pymol -cq -p", 'w+') do |pipe|
    > >>     pipe.puts "load file.pdb, mymodel\n"
    > >>     pipe.puts "run my_script.py"
    > >>     pipe.puts "my_script mymodel"

    >
    > >>     # IO.select([pipe]) # you shouldn't need to uncomment this...

    >
    > >>     output = pipe.read
    > >>   end

    >
    > >> --
    > >> Eric Wong

    >
    > > both of those hang indefinitely.  It doesn't hang if I add the line:

    >
    > >   pipe.puts "quit"

    >
    > > but that doesn't give me all the output either.  So maybe pymol
    > > behaves differently than many commandline programs?

    >
    > What does the documentation of "pymol" say?
    >
    >
    >
    > > Here is what I've worked out for the time being.  It opens a thread
    > > for reading the output and asks if there is any output every 1/2
    > > second.  Once we go 1/2 second without output we kill the thread.

    >
    > > my_string = ""
    > > Open3.popen3("pymol -cq -p") do |si, so, se|
    > >  si.puts "load file.pdb, mymodel\n"
    > >  si.puts "run my_script.py\n"
    > >  si.puts "myscript mymodel\n"

    >
    > >  forstdout = Thread.new do
    > >    Thread.current['lines'] = []
    > >    while line = so.gets
    > >      Thread.current['lines'] << line
    > >    end
    > >  end

    >
    > You don't need the thread local for this to work.  Note there is also
    > Thread#value!
    >
    > How does this work:
    >
    > output = IO.popen("pymol -cq -p", 'w+') do |pipe|
    >   reader = Thread.new { pipe.to_a }
    >
    >   pipe.puts "load file.pdb, mymodel"
    >   pipe.puts "run my_script.py"
    >   # this command will generate a whole bunch of output to stdout
    >   pipe.puts "my_script mymodel"
    >
    >   pipe.close_write
    >   reader.value
    > end
    >
    > Kind regards
    >
    > robert
    >
    > --
    > remember.guy do |as, often| as.you_can - without endhttp://blog.rubybestpractices.com/
    bwv549, Dec 17, 2009
    #6
  7. On 17.12.2009 06:40, bwv549 wrote:
    > @Robert: I've looked through a lot of pymol docs and haven't found
    > anything on it yet, will post if I do.
    >
    > The solution you post looks very intriguing. It runs fine, but closes
    > before I get back the output I need.


    Frankly, I don't see how this can be: the reader thread should read
    *everthing* that comes out of the pipe and only stops if there is EOF.
    The only explanation I have so far is that maybe some CTRL-D or other
    EOF character is sent through the pipe making the reader think it has
    reached EOF.

    > It seems like there must be some
    > kind of a pause to let pymol start spitting out output. I may play
    > around with it some to see if I can get it working.


    Even if there is a pause the reader must wait. There must be something
    else happening. Maybe try to do it on the shell and use 'od' to see
    what is sent across the pipe.

    > @Eric: this solution works nicely and returns all the expected
    > output. I will be using it unless something better comes along.


    Good that you have a working solution. However, what you state worries
    me a bit. I believe we might not have identified the root cause of the
    issue.

    > Thanks to both of you for your excellent input on this. --John


    You're welcome!

    Kind regards

    robert

    --
    remember.guy do |as, often| as.you_can - without end
    http://blog.rubybestpractices.com/
    Robert Klemme, Dec 17, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. lee, wonsun
    Replies:
    1
    Views:
    478
    Jack Klein
    Nov 2, 2004
  2. Tom Brown
    Replies:
    0
    Views:
    455
    Tom Brown
    Sep 22, 2005
  3. skunkwerk

    popen pipe limit

    skunkwerk, Apr 8, 2008, in forum: Python
    Replies:
    2
    Views:
    507
    skunkwerk
    Apr 10, 2008
  4. schickb

    Popen pipe hang

    schickb, May 13, 2008, in forum: Python
    Replies:
    0
    Views:
    289
    schickb
    May 13, 2008
  5. Replies:
    1
    Views:
    215
    Ben Morrow
    Jun 2, 2004
Loading...

Share This Page