Decompressing gzip over FTP

[SeanMon]

Is there a way to decompress a large (2GB) gzipped file being
retrieved over FTP on the fly?

I'm using ftplib.FTP to open a connection to a remote server, and I
have had no success connecting retrbinary to gzip without using an
intermediate file.

Is there any way to get a file-like object describing the remote file,
or a way to utilize gzip's decompression without providing it a file-
like object?

Thanks,
Sean

 

[SeanMon]

gzip is really just a file format. In order to work with compressed
streams you should use the low level zlib module.

http://docs.python.org/library/zlib.html#module-zlib

Have fun!

Christian

Unfortunately, the file on the server is a gzip file, so I cannot
simply pipe the contents into a zlib.decompressobj (which does have
the ability to decompress in chunks, as I wish gzip did!).

Thanks,
Sean

 

[Albert Hopkins]

Is there a way to decompress a large (2GB) gzipped file being
retrieved over FTP on the fly?

I'm using ftplib.FTP to open a connection to a remote server, and I
have had no success connecting retrbinary to gzip without using an
intermediate file.

Is there any way to get a file-like object describing the remote file,
or a way to utilize gzip's decompression without providing it a file-
like object?

I tried to solve your problem with StringIO and .truncate(). I also
tried to solve it with us.pipe(), but was unsuccessful with either. The
problem is the gzip module uses .seek() which you can't do with pipes...

You could probably get away with just using the compress module (as zlib
does) but you will have to deal with handling the headers, footers, CRC
checks, etc. on your own.

The cheap alternative is to just have your script open a pipe to "gunzip
-c" and read/write from that pipe.

hth,
-a