os.popen and broken pipes

P

Philipp Pagel

Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()


This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?

cu
Philipp
 
A

Antoon Pardon

Hi Pythoneers,

I need to process a large number of files which have been packed by the
UNIX compress tool (*.Z files). As I am not aware of a compress
equivalent of the gzip, zipfile or bzip2 modules, I thought I'd use the
uncompress or zcat commands directly to deal with the files:

for filename in file_list:
file = os.popen('uncompress -c '+filename, 'r')
do_something(file)
file.close()


This works fine for some files but results in

'write error onstdout: Broken pipe'

emitted by uncompress for others. Using zcat instead of uncompress
changes the wording of the error message but not the result.

I tried to give popen a large bufsize argument but that didn't really
help.

Probably I'm overlooking something obvious here but right now I can't
see it. Any hints?

As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.
 
P

Philipp Pagel

As far as I can tell, your do_something doesn't consume the entire file.
So you close the file prematurly, which results in the uncompress/zcat
program trying to write to a pipe that is closed on the otherside,
giving you the above message.

You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.

Thanks!

cu
Philipp
 
P

Philipp Pagel

Chris said:
It could easily be the 2gig file size limitation, how large are the
extracts?

The files are much smaller than that, so that's not the issue.
Anyway, Antoon pointed me in the right direction.

Thanks for the help
Philipp
 
D

Donn Cave

Philipp Pagel said:
You are right: some of the files do not fulfill certain
critereia causing so_somehting() to return before the entire file is
processed.

Most programming environments don't have this problem, though.

If you like, your program can undo what Python does:

signal.signal(signal.SIGPIPE, signal.SIG_DFL)
for filename in file_list:
...

Then it will work as if you had written it in C, or awk
or whatever.

Donn Cave, (e-mail address removed)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Broken Pipe 4
compiling perl 5.8.7 on Solaris 8 3
REQ: Perl 5.8.3 on OpenBSD 3

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,899
Latest member
RodneyMcAu

Latest Threads

Top