F
Fredrik Lundh
Claudio said:What started as a simple test if it is better to load uncompressed data
directly from the harddisk or
load compressed data and uncompress it (Windows XP SP 2, Pentium4 3.0 GHz
system with 3 GByte RAM)
seems to show that none of the in Python available compression libraries
really works for large sized
(i.e. 500 MByte) strings.
Test the provided code and see yourself.
At least on my system:
zlib fails to decompress raising a memory error
pylzma fails to decompress running endlessly consuming 99% of CPU time
bz2 fails to compress running endlessly consuming 99% of CPU time
The same works with a 10 MByte string without any problem.
So what? Is there no compression support for large sized strings in Python?
you're probably measuring windows' memory managment rather than the com-
pression libraries themselves (Python delegates all memory allocations >256 bytes
to the system).
I suggest using incremental (streaming) processing instead; from what I can tell,
all three libraries support that.
</F>