How to perform a nonblocking read from a process

[rdabane]

Hi,
I'm trying to perform following operation from inside the python
script
1. Open a shell ( start a process )
2. Send command1 to the process
3. Get output from the process
4. Send command2 to the process
5. Get output from the process
.......


Following is sample code :

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_newlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n')
o,e = p2.stdout.readline()
print o,e


It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

All the help is appreciated ..

Thanks,
-Rahul.

 

[sturlamolden]

It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

Threads are the simplest remedy for blocking i/o.

 

[rdabane]

Threads are the simplest remedy for blocking i/o.


Threads are the simplest remedy for blocking i/o.

I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...

 

[sturlamolden]

I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...

Are you planning on doing thousands of simultaneous asynchronous I/O
calls? How many processes are you communicating with? Take a look at
the twisted framwork (twistedmatrix.org) or perhaps i/o completion
ports on Windows (e.g. win32file.CreateIoCompletionPort in PyWin32).
But if you need that king of scalability, I suggest you start by
reconsidering the design.

If you are communicating with only one process, you don't need
thousands of threads, just one. Keep the thread idle until you need it
again. Here is a simple worker thread that allows you to do multiple,
different function calls in a background thread (you can shut it down
by passing None to setTask).


import threading
import Queue

class WorkerThread(threading.Thread):

def __init__(self):
self.taskQueue = Queue.Queue(0)
self.resQueue = Queue.Queue(0)
self.setDaemon(True)

def run(self):
while 1:
fun, args, kwarg = self.taskQueue.get()
if (fun is None): break
self.resQueue.put(fun(*args,**kwarg))

def setTask(self, fun, *args, **kwarg):
self.taskQueue.set((fun, args, kwarg))

def getResult(self):
return self.resQueue.get()

def probeResult(self):
return not self.resQueue.empty()


Rewrite of your examples:

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_newlines=True)
worker = WorkerThread()
for i in range(10):
worker.setTask(p2.stdin.write, 'print 10'+'\n')
worker.setTask(p2.stdout.readline)
worker.getResult() # wait for write to finish
o,e = worker.getResult() # wait for read to finish
print o,e

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_newlines=True)
writer = WorkerThread()
reader = WorkerThread()
for i in range(10):
writer.setTask(p2.stdin.write, 'print 10'+'\n')
reader.setTask(p2.stdout.readline)
o,e = reader.getResult() # wait for read to finish
print o,e