Moving to subprocess from os.popen--pipe closes prematurely

[Kevin Walzer]

Hello,

I'm trying to move from os.popen to using subprocess, and I'm having
trouble with the pipe suddenly closing.

My old code looked something like this:

#build and format a list of all packages
def getAllPackages(self):

self.masterlist=[]

for self.catname in self.catlist:
time.sleep(.3)
self.getpackages = os.popen('%s list --section=%s' %
(self.finkpath, self.catname), 'r', os.O_NONBLOCK)
for line in self.getpackages:
newline = line.split('\t')
rawcat = newline[0]
if rawcat == '(i)':
firstcat=rawcat.replace('(i)', 'outdated')
elif rawcat == ' i ':
firstcat=rawcat.replace('i', 'current')
elif rawcat == ' p ':
firstcat=rawcat.replace('p', 'provided')
else:
firstcat = rawcat
self.packagelist = (firstcat, newline[1], newline[2],
self.catname, newline[3].strip('\n'))
self.masterlist.append(self.packagelist)


Using the time.sleep function and adding the os.O_NONBLOCK to the
os.popen call let the data stream come through uninterrupted.

Here is the comparable code using subprocess:

def getAllPackages(self):

self.masterlist=[]
self.showProgress()
self.status.set('Getting all packages')
self.update()
for self.catname in self.catlist:
self.update()
self.status.set('Getting all packages by category: %s' %
self.catname)
self.getpackages = Popen('%s list --section=%s' %
(self.finkpath, self.catname), shell=True, bufsize=0, stdout=PIPE).stdout
time.sleep(5)
fcntl.fcntl(self.getpackages, fcntl.F_SETFL, os.O_NONBLOCK)
print "getting %s" % self.catname
self.update()
for line in self.getpackages:
print line
time.sleep(.1)
self.update()

newline = line.split('\t')
rawcat = newline[0]
if rawcat == '(i)':
firstcat=rawcat.replace('(i)', 'outdated')
elif rawcat == ' i ':
firstcat=rawcat.replace('i', 'current')
elif rawcat == ' p ':
firstcat=rawcat.replace('p', 'provided')
else:
firstcat = rawcat
self.packagelist = (firstcat, newline[1], newline[2],
self.catname, newline[3].strip('\n'))


I am using the fcntl call to set the pipe to nonblocking mode, and am
trying to time.sleep to give the buffer time to get some data. However,
this code crashes with the error "IOError: [Errno 35] Resource
temporarily unavailable."

Any advice to get the subprocess bits working so that the buffer does
not close prematurely would be appreciated.

Thank you,
Kevin

 

[Martin P. Hellwig]

Hello,

I'm trying to move from os.popen to using subprocess, and I'm having
trouble with the pipe suddenly closing.

My old code looked something like this:

<cut code and rest>

Hi Kevin,

You could try something more like:

>>> import subprocess
>>> cmd = subprocess.Popen([executable_path, executable_options], stdout=subprocess.PIPE, stdout=subprocess.PIPE)
>>> std_out, std_err = cmd.communicate(standard_in_value)

std_out is a string though, you probably need to split it on newline to
get the same sort of list and since it is buffered in memory you
probably don't want to use .communicate if you expect megabytes of data
back.