Issue combining gzip and subprocess

[Iwan Vosloo]

Hi there,

We tried to gzip the output of a shell command, but this results in a
strange error: the resulting file seems to be the concatenation of the
plaintext file with the zipped content.

For example:

f = gzip.open(filename, 'w')
subprocess.check_call(['ls','-la'], stdout=f)
f.close()

Using a normal file works as expected, but a GzipFile results in a file
containing what looks like the unzipped data, followed by the zipped
data.

I suspect this may have something to do with limitations of GzipFile
such as it not implementing truncate().

Does anyone have an explanation / can you suggest a nice solution for
doing what we are trying to do?

Regards
- Iwan

 

[Piet van Oostrum]

IV> Hi there,
IV> We tried to gzip the output of a shell command, but this results in a
IV> strange error: the resulting file seems to be the concatenation of the
IV> plaintext file with the zipped content.

IV> For example:

IV> f = gzip.open(filename, 'w')
IV> subprocess.check_call(['ls','-la'], stdout=f)
IV> f.close()

IV> Using a normal file works as expected, but a GzipFile results in a file
IV> containing what looks like the unzipped data, followed by the zipped
IV> data.

IV> I suspect this may have something to do with limitations of GzipFile
IV> such as it not implementing truncate().

IV> Does anyone have an explanation / can you suggest a nice solution for
IV> doing what we are trying to do?

stdout (and the others) must be None, PIPE or a real file object or file
descriptor, not a file like object. In your case the solution would be
to use PIPE, and then read the output and write in to the GzipFile
yourself.

f = gzip.open(filename, 'w')
proc = subprocess.Popen(['ls','-la'], stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line: break
f.write(line)
f.close()

 

[Piet van Oostrum]

...
f = gzip.open(filename, 'w')
proc = subprocess.Popen(['ls','-la'], stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line: break
f.write(line)
f.close()

SDD> Or even:
SDD> proc = subprocess.Popen(['ls','-la'], stdout=subprocess.PIPE)
SDD> with gzip.open(filename, 'w') as dest:
SDD> for line in iter(proc.stdout, ''):
SDD> f.write(line)

If it would work.

1) with gzip... is not supported in Python < 3.1
2) for line in iter(proc.stdout), i.e. no second argument.
3) dest <==> f should be the same identifier.

Lesson: if you post code either:
- test it and copy verbatim from your test, or
- write a disclaimer