Re: read large zip file

Discussion in 'Python' started by Gabriel Genellina, Apr 6, 2008.

  1. En Sun, 06 Apr 2008 19:20:31 -0300, Brian Blais <>
    escribió:

    > I need to read a series of large zipfiles (which only contain one
    > large text file), and I noticed that the zipfile module:
    >
    > 1) has a read method which isn't an iterator, and returns the entire
    > file selected all at once
    > 2) has no readlines method, and no obvious way to implement one
    >
    > Is there a way to stream an unzip, so it behaves more like a file?


    Use the module from the 2.6 version; it appears to work fine even on
    Python 2.4 (see this thread
    http://groups.google.com/group/comp.lang.python/browse_thread/thread/71c4890cefac82aa/
    )

    --
    Gabriel Genellina
    Gabriel Genellina, Apr 6, 2008
    #1
    1. Advertising

  2. Gabriel Genellina

    John Nagle Guest

    Gabriel Genellina wrote:
    > En Sun, 06 Apr 2008 19:20:31 -0300, Brian Blais <>
    > escribió:
    >
    >> I need to read a series of large zipfiles (which only contain one
    >> large text file), and I noticed that the zipfile module:
    >>
    >> 1) has a read method which isn't an iterator, and returns the entire
    >> file selected all at once
    >> 2) has no readlines method, and no obvious way to implement one
    >>
    >> Is there a way to stream an unzip, so it behaves more like a file?

    >
    > Use the module from the 2.6 version; it appears to work fine even on
    > Python 2.4 (see this thread
    > http://groups.google.com/group/comp.lang.python/browse_thread/thread/71c4890cefac82aa/
    > )


    It's easier than that:

    fd = gzip.open(filename, 'rb')
    for line in fd :
    processline(line)

    This works even in Python 2.4. I use this routinely for processing big
    log files.

    John Nagle
    John Nagle, Apr 8, 2008
    #2
    1. Advertising

  3. En Tue, 08 Apr 2008 00:10:01 -0300, John Nagle <>
    escribió:
    > Gabriel Genellina wrote:
    >> En Sun, 06 Apr 2008 19:20:31 -0300, Brian Blais <>
    >> escribió:
    >>
    >>> I need to read a series of large zipfiles (which only contain one
    >>> large text file), and I noticed that the zipfile module:

    >>
    >> Use the module from the 2.6 version; it appears to work fine even on
    >> Python 2.4 (see this thread

    >
    > It's easier than that:
    >
    > fd = gzip.open(filename, 'rb')
    > for line in fd :
    > processline(line)
    >
    > This works even in Python 2.4. I use this routinely for processing big
    > log files.


    That works for gzipped files, but the OP said "zipfiles" which aren't the
    same thing. It might be a generic term too - we'll have to wait until he
    gives any feedback...

    --
    Gabriel Genellina
    Gabriel Genellina, Apr 8, 2008
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    0
    Views:
    1,716
  2. abcd
    Replies:
    7
    Views:
    510
    Edward Elliott
    May 1, 2006
  3. majj81
    Replies:
    0
    Views:
    330
    majj81
    May 31, 2006
  4. Chris Lambacher

    Re: Add file to zip, or replace file in zip

    Chris Lambacher, Jun 1, 2006, in forum: Python
    Replies:
    0
    Views:
    597
    Chris Lambacher
    Jun 1, 2006
  5. Kevin Ar18

    Unable to read large files from zip

    Kevin Ar18, Aug 29, 2007, in forum: Python
    Replies:
    2
    Views:
    517
    David Bolen
    Aug 29, 2007
Loading...

Share This Page