How to send a var to stdin of an external software

[Benjamin Watine]

Hi the list,

I need to send a var to stdin of an external soft ("cat" command for
example).

How can I do this ? I would like a function like that :

theFunction ('cat -', stdin=myVar)

I don't need to get any return value.

Another related question : Is there's a limitation of var size ? I would
have var up to 10 MB.

Thanks !

Ben

 

[Marko Rauhamaa]

How can I do this ? I would like a function like that :

theFunction ('cat -', stdin=myVar)

Another related question : Is there's a limitation of var size ? I
would have var up to 10 MB.

import subprocess
myVar = '*' * 10000000
cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()


Marko

 

[Sion Arrowsmith]

How can I do this ? I would like a function like that :

theFunction ('cat -', stdin=myVar)

I don't need to get any return value.

http://docs.python.org/lib/node534.html says this is spelt

myVar = subprocess.Popen(["cat", "-"], stdout=subprocess.PIPE).communicate()[0]

(Probably not obvious how to find this if you've not come across the
backtick notation in shell or Perl.)

 

[Benjamin Watine]

Marko Rauhamaa a écrit :

import subprocess
myVar = '*' * 10000000
cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()


Marko

Thank you Marko, it's exactly what I need.

And if somebody need it : to get the stdout in a var (myNewVar), not in
the shell :

cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
stdout=subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = cat.stdout.read()

Is it correct ?

Ben

 

[Bryan Olson]

And if somebody need it : to get the stdout in a var (myNewVar), not in
the shell :

cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
stdout=subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = cat.stdout.read()

Is it correct ?

No, not really. It is prone to deadlock. The external program might
work by iteratively reading a little input and writing a little
output, as 'cat' almost surely does. If the size of myVar exceeds
the buffer space in cat and the pipes, you get stuck.

Your Python program can block at "cat.stdin.write(myVar)", waiting
for cat to read from its input pipe, while cat blocks at a write
to its output stream, waiting for you to start reading and freeing
up buffer space. Pipe loops are tricky business.

Popular solutions are to make either the input or output stream
a disk file, or to create another thread (or process) to be an
active reader or writer.

 

[Bryan Olson]

[...] Pipe loops are tricky business.

Popular solutions are to make either the input or output stream
a disk file, or to create another thread (or process) to be an
active reader or writer.

Or asynchronous I/O. On Unix-like systems, you can select() on
the underlying file descriptors. (MS-Windows async mechanisms are
not as well exposed by the Python standard library.)

 

[Benjamin Watine]

Bryan Olson a écrit :

[...] Pipe loops are tricky business.

Popular solutions are to make either the input or output stream
a disk file, or to create another thread (or process) to be an
active reader or writer.

Or asynchronous I/O. On Unix-like systems, you can select() on
the underlying file descriptors. (MS-Windows async mechanisms are
not as well exposed by the Python standard library.)

Hi Bryan

Thank you so much for your advice. You're right, I just made a test with
a 10 MB input stream, and it hangs exactly like you said (on
cat.stdin.write(myStdin))...

I don't want to use disk files. In reality, this script was previously
done in bash using disk files, but I had problems with that solution
(the files wasn't always cleared, and sometimes, I've found a part of
previous input at the end of the next input.)

That's why I want to use python, just to not use disk files.

Could you give me more information / examples about the two solutions
you've proposed (thread or asynchronous I/O) ?

Thank you !

Ben

 

[Floris Bruynooghe]

Bryan Olson a écrit :

[...] Pipe loops are tricky business.
Popular solutions are to make either the input or output stream
a disk file, or to create another thread (or process) to be an
active reader or writer.

Or asynchronous I/O. On Unix-like systems, you can select() on
the underlying file descriptors. (MS-Windows async mechanisms are
not as well exposed by the Python standard library.)

Hi Bryan

Thank you so much for your advice. You're right, I just made a test with
a 10 MB input stream, and it hangs exactly like you said (on
cat.stdin.write(myStdin))...

I don't want to use disk files. In reality, this script was previously
done in bash using disk files, but I had problems with that solution
(the files wasn't always cleared, and sometimes, I've found a part of
previous input at the end of the next input.)

That's why I want to use python, just to not use disk files.

Could you give me more information / examples about the two solutions
you've proposed (thread or asynchronous I/O) ?

The source code of the subprocess module shows how to do it with
select IIRC. Look at the implementation of the communicate() method.

Regards
Floris

 

[bryanjugglercryptographer]

The source code of the subprocess module shows how to do it with
select IIRC. Look at the implementation of the communicate() method.

And here's a thread example, based on Benjamin's code:

import subprocess
import thread

def readtobox(pipe, box):
box.append(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

box = []
thread.start_new_thread(readtobox, (cat.stdout, box))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = box[0]

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."

 

[bryanjugglercryptographer]

And here's a thread example, based on Benjamin's code:

[...]

Doh! Race condition. Make that:

import subprocess
import thread
import Queue

def readtoq(pipe, q):
q.put(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

q = Queue.Queue()
thread.start_new_thread(readtoq, (cat.stdout, q))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = q.get()

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."

 

[Benjamin Watine]

(e-mail address removed) a écrit :

And here's a thread example, based on Benjamin's code:

[...]

Doh! Race condition. Make that:

import subprocess
import thread
import Queue

def readtoq(pipe, q):
q.put(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

q = Queue.Queue()
thread.start_new_thread(readtoq, (cat.stdout, q))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = q.get()

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."

Great, it works, thank you Bryan !

Could you explain me why you use a queue instead of a simple array for
getting the piped var ?

Regards,

Ben

 

[Bryan Olson]

(e-mail address removed) a écrit :

And here's a thread example, based on Benjamin's code:

[...]

Doh! Race condition. Make that:

import subprocess
import thread
import Queue

def readtoq(pipe, q):
q.put(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

q = Queue.Queue()
thread.start_new_thread(readtoq, (cat.stdout, q))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = q.get()

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."

Great, it works, thank you Bryan !

Could you explain me why you use a queue instead of a simple array for
getting the piped var ?

The call to q.get() will block until an item is in the queue.
At that point in the program, we had already waited for cat to
terminate:

cat.wait()
myNewVar = q.get()

But passing cat.wait() does not imply that our own thread has
already read all of cat's output and put it in some destination
object. Data could still be in transit.

My first version, subsequently titled, "Doh! Race condition,"
worked in all of several runs of its built-in test. Doesn't
make it right.

 

[Benjamin Watine]

Bryan Olson a écrit :

(e-mail address removed) a écrit :

I wrote:
And here's a thread example, based on Benjamin's code:
[...]

Doh! Race condition. Make that:

import subprocess
import thread
import Queue

def readtoq(pipe, q):
q.put(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

q = Queue.Queue()
thread.start_new_thread(readtoq, (cat.stdout, q))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = q.get()

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."

Great, it works, thank you Bryan !

Could you explain me why you use a queue instead of a simple array for
getting the piped var ?

The call to q.get() will block until an item is in the queue.
At that point in the program, we had already waited for cat to
terminate:

cat.wait()
myNewVar = q.get()

But passing cat.wait() does not imply that our own thread has
already read all of cat's output and put it in some destination
object. Data could still be in transit.

My first version, subsequently titled, "Doh! Race condition,"
worked in all of several runs of its built-in test. Doesn't
make it right.

OK, so if I understand well what you said, using queue allow to be sure
that the data is passed in totality before coninuing with next
instruction. That make sense.
Using thread and queue seems to be very more slow than using files
redirection with bash. I'll see if it make a great load average and/or
I/O time.

Thanks again for your help Bryan.

Ben

 

[Bryan Olson]

OK, so if I understand well what you said, using queue allow to be sure
that the data is passed in totality before coninuing with next
instruction. That make sense.
Right.

Using thread and queue seems to be very more slow than using files
redirection with bash. I'll see if it make a great load average and/or
I/O time.

Hmmm... you might try increasing the buffer size.