How to send a var to stdin of an external software

Discussion in 'Python' started by Benjamin Watine, Mar 13, 2008.

  1. Hi the list,

    I need to send a var to stdin of an external soft ("cat" command for
    example).

    How can I do this ? I would like a function like that :

    theFunction ('cat -', stdin=myVar)

    I don't need to get any return value.

    Another related question : Is there's a limitation of var size ? I would
    have var up to 10 MB.

    Thanks !

    Ben
    Benjamin Watine, Mar 13, 2008
    #1
    1. Advertising

  2. Benjamin Watine <>:

    > How can I do this ? I would like a function like that :
    >
    > theFunction ('cat -', stdin=myVar)
    >
    > Another related question : Is there's a limitation of var size ? I
    > would have var up to 10 MB.


    import subprocess
    myVar = '*' * 10000000
    cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
    cat.stdin.write(myVar)
    cat.stdin.close()
    cat.wait()


    Marko

    --
    Marko Rauhamaa mailto: http://pacujo.net/marko/
    Marko Rauhamaa, Mar 13, 2008
    #2
    1. Advertising

  3. Benjamin Watine <> wrote:
    >How can I do this ? I would like a function like that :
    >
    > theFunction ('cat -', stdin=myVar)
    >
    >I don't need to get any return value.


    http://docs.python.org/lib/node534.html says this is spelt

    myVar = subprocess.Popen(["cat", "-"], stdout=subprocess.PIPE).communicate()[0]

    (Probably not obvious how to find this if you've not come across the
    backtick notation in shell or Perl.)

    --
    \S -- -- http://www.chaos.org.uk/~sion/
    "Frankly I have no feelings towards penguins one way or the other"
    -- Arthur C. Clarke
    her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
    Sion Arrowsmith, Mar 13, 2008
    #3
  4. Marko Rauhamaa a écrit :
    > Benjamin Watine <>:
    >
    >> How can I do this ? I would like a function like that :
    >>
    >> theFunction ('cat -', stdin=myVar)
    >>
    >> Another related question : Is there's a limitation of var size ? I
    >> would have var up to 10 MB.

    >
    > import subprocess
    > myVar = '*' * 10000000
    > cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
    > cat.stdin.write(myVar)
    > cat.stdin.close()
    > cat.wait()
    >
    >
    > Marko
    >


    Thank you Marko, it's exactly what I need.

    And if somebody need it : to get the stdout in a var (myNewVar), not in
    the shell :

    cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
    stdout=subprocess.PIPE)
    cat.stdin.write(myVar)
    cat.stdin.close()
    cat.wait()
    myNewVar = cat.stdout.read()

    Is it correct ?

    Ben
    Benjamin Watine, Mar 13, 2008
    #4
  5. Benjamin Watine

    Bryan Olson Guest

    Benjamin Watine wrote:
    > And if somebody need it : to get the stdout in a var (myNewVar), not in
    > the shell :
    >
    > cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
    > stdout=subprocess.PIPE)
    > cat.stdin.write(myVar)
    > cat.stdin.close()
    > cat.wait()
    > myNewVar = cat.stdout.read()
    >
    > Is it correct ?


    No, not really. It is prone to deadlock. The external program might
    work by iteratively reading a little input and writing a little
    output, as 'cat' almost surely does. If the size of myVar exceeds
    the buffer space in cat and the pipes, you get stuck.

    Your Python program can block at "cat.stdin.write(myVar)", waiting
    for cat to read from its input pipe, while cat blocks at a write
    to its output stream, waiting for you to start reading and freeing
    up buffer space. Pipe loops are tricky business.

    Popular solutions are to make either the input or output stream
    a disk file, or to create another thread (or process) to be an
    active reader or writer.


    --
    --Bryan
    Bryan Olson, Mar 13, 2008
    #5
  6. Benjamin Watine

    Bryan Olson Guest

    I wrote:
    > [...] Pipe loops are tricky business.
    >
    > Popular solutions are to make either the input or output stream
    > a disk file, or to create another thread (or process) to be an
    > active reader or writer.


    Or asynchronous I/O. On Unix-like systems, you can select() on
    the underlying file descriptors. (MS-Windows async mechanisms are
    not as well exposed by the Python standard library.)


    --
    --Bryan
    Bryan Olson, Mar 13, 2008
    #6
  7. Bryan Olson a écrit :
    > I wrote:
    >> [...] Pipe loops are tricky business.
    >>
    >> Popular solutions are to make either the input or output stream
    >> a disk file, or to create another thread (or process) to be an
    >> active reader or writer.

    >
    > Or asynchronous I/O. On Unix-like systems, you can select() on
    > the underlying file descriptors. (MS-Windows async mechanisms are
    > not as well exposed by the Python standard library.)
    >


    Hi Bryan

    Thank you so much for your advice. You're right, I just made a test with
    a 10 MB input stream, and it hangs exactly like you said (on
    cat.stdin.write(myStdin))...

    I don't want to use disk files. In reality, this script was previously
    done in bash using disk files, but I had problems with that solution
    (the files wasn't always cleared, and sometimes, I've found a part of
    previous input at the end of the next input.)

    That's why I want to use python, just to not use disk files.

    Could you give me more information / examples about the two solutions
    you've proposed (thread or asynchronous I/O) ?

    Thank you !

    Ben
    Benjamin Watine, Mar 14, 2008
    #7
  8. On Mar 14, 11:37 am, Benjamin Watine <> wrote:
    > Bryan Olson a écrit :
    >
    > > I wrote:
    > >> [...] Pipe loops are tricky business.

    >
    > >> Popular solutions are to make either the input or output stream
    > >> a disk file, or to create another thread (or process) to be an
    > >> active reader or writer.

    >
    > > Or asynchronous I/O. On Unix-like systems, you can select() on
    > > the underlying file descriptors. (MS-Windows async mechanisms are
    > > not as well exposed by the Python standard library.)

    >
    > Hi Bryan
    >
    > Thank you so much for your advice. You're right, I just made a test with
    > a 10 MB input stream, and it hangs exactly like you said (on
    > cat.stdin.write(myStdin))...
    >
    > I don't want to use disk files. In reality, this script was previously
    > done in bash using disk files, but I had problems with that solution
    > (the files wasn't always cleared, and sometimes, I've found a part of
    > previous input at the end of the next input.)
    >
    > That's why I want to use python, just to not use disk files.
    >
    > Could you give me more information / examples about the two solutions
    > you've proposed (thread or asynchronous I/O) ?


    The source code of the subprocess module shows how to do it with
    select IIRC. Look at the implementation of the communicate() method.

    Regards
    Floris
    Floris Bruynooghe, Mar 14, 2008
    #8
  9. Benjamin Watine

    Guest

    Floris Bruynooghe wrote:
    > Benjamin Watine wrote:
    > > Could you give me more information / examples about the two solutions
    > > you've proposed (thread or asynchronous I/O) ?

    >
    > The source code of the subprocess module shows how to do it with
    > select IIRC. Look at the implementation of the communicate() method.


    And here's a thread example, based on Benjamin's code:

    import subprocess
    import thread

    def readtobox(pipe, box):
    box.append(pipe.read())

    cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
    stdout=subprocess.PIPE)

    myVar = str(range(1000000)) # arbitrary test data.

    box = []
    thread.start_new_thread(readtobox, (cat.stdout, box))
    cat.stdin.write(myVar)
    cat.stdin.close()
    cat.wait()
    myNewVar = box[0]

    assert myNewVar == myVar
    print len(myNewVar), "bytes piped around."


    --
    --Bryan
    , Mar 14, 2008
    #9
  10. Benjamin Watine

    Guest

    I wrote:
    > And here's a thread example, based on Benjamin's code:

    [...]

    Doh! Race condition. Make that:

    import subprocess
    import thread
    import Queue

    def readtoq(pipe, q):
    q.put(pipe.read())

    cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
    stdout=subprocess.PIPE)

    myVar = str(range(1000000)) # arbitrary test data.

    q = Queue.Queue()
    thread.start_new_thread(readtoq, (cat.stdout, q))
    cat.stdin.write(myVar)
    cat.stdin.close()
    cat.wait()
    myNewVar = q.get()

    assert myNewVar == myVar
    print len(myNewVar), "bytes piped around."


    --
    --Bryan
    , Mar 14, 2008
    #10
  11. a écrit :
    > I wrote:
    >> And here's a thread example, based on Benjamin's code:

    > [...]
    >
    > Doh! Race condition. Make that:
    >
    > import subprocess
    > import thread
    > import Queue
    >
    > def readtoq(pipe, q):
    > q.put(pipe.read())
    >
    > cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
    > stdout=subprocess.PIPE)
    >
    > myVar = str(range(1000000)) # arbitrary test data.
    >
    > q = Queue.Queue()
    > thread.start_new_thread(readtoq, (cat.stdout, q))
    > cat.stdin.write(myVar)
    > cat.stdin.close()
    > cat.wait()
    > myNewVar = q.get()
    >
    > assert myNewVar == myVar
    > print len(myNewVar), "bytes piped around."
    >
    >
    > --
    > --Bryan
    >


    Great, it works, thank you Bryan !

    Could you explain me why you use a queue instead of a simple array for
    getting the piped var ?

    Regards,

    Ben
    Benjamin Watine, Mar 17, 2008
    #11
  12. Benjamin Watine

    Bryan Olson Guest

    Benjamin Watine wrote:
    > a écrit :
    >> I wrote:
    >>> And here's a thread example, based on Benjamin's code:

    >> [...]
    >>
    >> Doh! Race condition. Make that:
    >>
    >> import subprocess
    >> import thread
    >> import Queue
    >>
    >> def readtoq(pipe, q):
    >> q.put(pipe.read())
    >>
    >> cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
    >> stdout=subprocess.PIPE)
    >>
    >> myVar = str(range(1000000)) # arbitrary test data.
    >>
    >> q = Queue.Queue()
    >> thread.start_new_thread(readtoq, (cat.stdout, q))
    >> cat.stdin.write(myVar)
    >> cat.stdin.close()
    >> cat.wait()
    >> myNewVar = q.get()
    >>
    >> assert myNewVar == myVar
    >> print len(myNewVar), "bytes piped around."


    >
    > Great, it works, thank you Bryan !
    >
    > Could you explain me why you use a queue instead of a simple array for
    > getting the piped var ?


    The call to q.get() will block until an item is in the queue.
    At that point in the program, we had already waited for cat to
    terminate:

    cat.wait()
    myNewVar = q.get()

    But passing cat.wait() does not imply that our own thread has
    already read all of cat's output and put it in some destination
    object. Data could still be in transit.

    My first version, subsequently titled, "Doh! Race condition,"
    worked in all of several runs of its built-in test. Doesn't
    make it right.


    --
    --Bryan
    Bryan Olson, Mar 21, 2008
    #12
  13. Bryan Olson a écrit :
    > Benjamin Watine wrote:
    >> a écrit :
    >>> I wrote:
    >>>> And here's a thread example, based on Benjamin's code:
    >>> [...]
    >>>
    >>> Doh! Race condition. Make that:
    >>>
    >>> import subprocess
    >>> import thread
    >>> import Queue
    >>>
    >>> def readtoq(pipe, q):
    >>> q.put(pipe.read())
    >>>
    >>> cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
    >>> stdout=subprocess.PIPE)
    >>>
    >>> myVar = str(range(1000000)) # arbitrary test data.
    >>>
    >>> q = Queue.Queue()
    >>> thread.start_new_thread(readtoq, (cat.stdout, q))
    >>> cat.stdin.write(myVar)
    >>> cat.stdin.close()
    >>> cat.wait()
    >>> myNewVar = q.get()
    >>>
    >>> assert myNewVar == myVar
    >>> print len(myNewVar), "bytes piped around."

    >
    >> Great, it works, thank you Bryan !
    >>
    >> Could you explain me why you use a queue instead of a simple array for
    >> getting the piped var ?

    >
    > The call to q.get() will block until an item is in the queue.
    > At that point in the program, we had already waited for cat to
    > terminate:
    >
    > cat.wait()
    > myNewVar = q.get()
    >
    > But passing cat.wait() does not imply that our own thread has
    > already read all of cat's output and put it in some destination
    > object. Data could still be in transit.
    >
    > My first version, subsequently titled, "Doh! Race condition,"
    > worked in all of several runs of its built-in test. Doesn't
    > make it right.
    >

    OK, so if I understand well what you said, using queue allow to be sure
    that the data is passed in totality before coninuing with next
    instruction. That make sense.
    Using thread and queue seems to be very more slow than using files
    redirection with bash. I'll see if it make a great load average and/or
    I/O time.

    Thanks again for your help Bryan.

    Ben
    Benjamin Watine, Mar 25, 2008
    #13
  14. Benjamin Watine

    Bryan Olson Guest

    Benjamin Watine wrote:
    > OK, so if I understand well what you said, using queue allow to be sure
    > that the data is passed in totality before coninuing with next
    > instruction. That make sense.


    Right.

    > Using thread and queue seems to be very more slow than using files
    > redirection with bash. I'll see if it make a great load average and/or
    > I/O time.


    Hmmm... you might try increasing the buffer size.


    --
    --Bryan
    Bryan Olson, Mar 26, 2008
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alvin Bruney

    Threads.. Session var lost, App var ok

    Alvin Bruney, Dec 2, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    349
    rooster575
    Dec 2, 2003
  2. thomson
    Replies:
    10
    Views:
    2,475
    Eliyahu Goldin
    Jun 20, 2005
  3. thomson
    Replies:
    0
    Views:
    361
    thomson
    Jun 20, 2005
  4. Fred
    Replies:
    3
    Views:
    314
    Alf P. Steinbach
    Aug 10, 2003
  5. Alex Vinokur
    Replies:
    4
    Views:
    463
    Ron Natalie
    Sep 24, 2003
Loading...

Share This Page