subprocess.Popen stalls

[psaffrey]

I'm building a bioinformatics application using the ipcress tool:

http://www.ebi.ac.uk/~guy/exonerate/ipcress.man.html

I'm using subprocess.Popen to execute ipcress, which takes a group of
files full of DNA sequences and returns some analysis on them. Here's
a code fragment:

cmd = "/usr/bin/ipcress ipcresstmp.txt --sequence /home/pzs/genebuilds/
human/*.fasta"
print "checking with ipcress using command", cmd
p = Popen(cmd, shell=True, bufsize=100, stdout=PIPE, stderr=PIPE)
retcode = p.wait()
if retcode != 0:
print "ipcress failed with error code:", retcode
raise Exception
output = p.stdout.read()

If I run the command at my shell, it finishes successfully. It takes
30 seconds - it uses 100% of one core and several hundred MB of memory
during this time. The output is 220KB of text.

However, running it through Python as per the above code, it stalls
after 5 seconds not using any processor at all. I've tried leaving it
for a few minutes with no change. If I interrupt it, it's at the
"retcode = p.wait()" line.

I've tried making the bufsize really large and that doesn't seem to
help. I'm a bit stuck - any suggestions? This same command has worked
fine on other ipcress runs. This one might generate more output than
the others, but 220KB isn't that much, is it?

Peter

Peter

 

[mk]

p = Popen(cmd, shell=True, bufsize=100, stdout=PIPE, stderr=PIPE)
output = p.stdout.read()

Better use communicate() method:

standardoutputstr, standarderrorstr = subprocess.communicate(...)

Never had any problem with subprocesses when using subprocess module in
this manner (well it's possible that standardoutputstr and/or
standarderrorstr fill up the memory, you get the idea).

Regards,
mk

 

[psaffrey]

Better use communicate() method:

Oh yes - it's right there in the documentation. That worked perfectly.

Many thanks,

Peter

 

[mk]

Oh yes - it's right there in the documentation. That worked perfectly.

What's also in the docs and I did not pay attention to before:

Note

The data read is buffered in memory, so do not use this method if the
data size is large or unlimited.


Regards,
mk