T
Tim Chase
I'm dealing with several large items that have been zipped up to
get quite impressive compression. However, uncompressed, they're
large enough to thrash my memory to swap and in general do bad
performance-related things. I'm trying to figure out how to
produce a file-like iterator out of the contents of such an item.
20
I need to be able to access multiple files within it, but I can
iterate over each one, only seeing small slices of the file.
Using the read() method triggers the volumnous read. Thus what I
have to do currently:
What I'm trying to figure out how to do is something like the
mythical:
to more efficiently handle the huge stream of data.
Am I missing something obvious? It seems like iterating over zip
contents would be a common thing to do (especially when compared
to reading the whole contents...I mean, they're zipped because
they're big!
Thanks for any pointers,
-tkc
get quite impressive compression. However, uncompressed, they're
large enough to thrash my memory to swap and in general do bad
performance-related things. I'm trying to figure out how to
produce a file-like iterator out of the contents of such an item.
20
I need to be able to access multiple files within it, but I can
iterate over each one, only seeing small slices of the file.
Using the read() method triggers the volumnous read. Thus what I
have to do currently:
What I'm trying to figure out how to do is something like the
mythical:
to more efficiently handle the huge stream of data.
Am I missing something obvious? It seems like iterating over zip
contents would be a common thing to do (especially when compared
to reading the whole contents...I mean, they're zipped because
they're big!
Thanks for any pointers,
-tkc