Open a command pipe for reading

R

Rodrick Brown

I have a fairly large file 1-2GB in size that I need to process line by line but I first need to convert the file to text using a 3rd party tool that prints the records also line by line.

I've tried using Popen to do this with no luck. I'm trying to simulate

/bin/foo myfile.dat

And as the records are being printed do some calculations.

pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

for data in pipe.readlines():
print data,

This operation blocks forever I'm guessing it's trying to process the entire file at once.

Sent from my iPhone 4.
 
S

Stefan Schwarzer

Hi Rodrick,

I have a fairly large file 1-2GB in size that I need to
process line by line but I first need to convert the file
to text using a 3rd party tool that prints the records
also line by line.

I've tried using Popen to do this with no luck. I'm trying
to simulate

/bin/foo myfile.dat

Is foo the 3rd-party conversion tool you've mentioned?

It would be good to have a bit more context, e. g. an actual
snippet from your code.
And as the records are being printed do some calculations.

pipe = Popen(exttool,shell=True,stdout=PIPE).stdout

for data in pipe.readlines():
print data,

This operation blocks forever I'm guessing it's trying to
process the entire file at once.

If you use `readlines` on a file object it reads the whole
file at once and returns a list of the lines (including line
end characters, by the way). What you probably want is

for line in pipe:
print line,

which reads and prints the file contents line by line.

Stefan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,830
Latest member
ZADIva7383

Latest Threads

Top