Reading just a few lines from a text file

Discussion in 'Python' started by tkpmep@hotmail.com, Aug 23, 2005.

  1. Guest

    I have a text file with many hundreds of lines of data. The data of
    interest to me, however, resides at the bottom of the file, in the last
    20 lines. Right now, I read the entire file and discard the stuff I
    don't need. I'd like to speed up my program by reading only the last 20
    lines. How do I do this?

    Thomas Philips
    , Aug 23, 2005
    #1
    1. Advertising

  2. rafi Guest

    wrote:
    > I have a text file with many hundreds of lines of data. The data of
    > interest to me, however, resides at the bottom of the file, in the last
    > 20 lines. Right now, I read the entire file and discard the stuff I
    > don't need. I'd like to speed up my program by reading only the last 20
    > lines. How do I do this?
    >
    > Thomas Philips
    >


    If you are using a Unix:

    tail -20 file.txt

    --
    rafi

    "Imagination is more important than knowledge."
    (Albert Einstein)
    rafi, Aug 23, 2005
    #2
    1. Advertising

  3. Guest

    I just did strace on "tail -20 <filename>". Apparently, it does seek to
    the end and reads enough data to cover 20 lines. I guess it is
    calculating this "size" by counting 20 new lines .You may try to do the
    same thing.

    Thanks,
    Raghu.
    , Aug 23, 2005
    #3
  4. Paul McGuire Guest

    Are you sure this is really slowing down your program? "Many hundreds
    of lines" is not nearly enough to start Python breathing hard. I have
    been really impressed with just how quickly Python is able to do file
    input and processing, zipping through whole megs of data in just
    seconds.

    How are you currently reading the file in? A character at a time?
    That *will* be slow. Try file.readlines(), or xreadlines(), and spin
    off the last 20 in the list.

    -- Paul
    Paul McGuire, Aug 23, 2005
    #4
  5. Guest

    Right now my code reads as follows:

    infile=file(FileName)
    for line in reversed(infile.readlines()): #Search from the bottom up
    if int(line.split()[0]) == MyDate:
    Data= float(line.split()[-1])
    break
    infile.close()

    I have to read about 10,000 files, each with data. I'm looking to speed
    up each individual file open/close cycle.

    Thomas Philips
    , Aug 23, 2005
    #5
  6. wrote:
    > Right now my code reads as follows:
    >
    > infile=file(FileName)
    > for line in reversed(infile.readlines()): #Search from the bottom up


    Not sure if python does some tricks here - but for me that seems to be
    uneccesary shuffling around of data. Better do

    for line in reversed(infile.readlines()[:-10]):
    ...


    Diez
    Diez B. Roggisch, Aug 23, 2005
    #6
  7. John Machin Guest

    wrote:
    > I have a text file with many hundreds of lines of data. The data of
    > interest to me, however, resides at the bottom of the file, in the last
    > 20 lines. Right now, I read the entire file and discard the stuff I
    > don't need. I'd like to speed up my program by reading only the last 20
    > lines. How do I do this?
    >
    > Thomas Philips
    >


    What discernible speed increase are you talking about? How long does it
    take to read the "many hundreds" of lines?

    For "many hundreds", IMHO it's not worth the bother, the complexity, the
    documentation, ...

    Just do this, it's about as fast as you'll get in pure python for a
    smallish file:

    def last_n_lines(filename, n):
    return open(filename, 'r').readlines()[-n:]

    For many hundreds of thousands of lines, one approach might be to open
    the file in binary mode, seek to the end of the file, then loop reading
    chunks backwards and unpacking the chunks until you've found 21 line
    terminators. Or perhaps 20 line separators :)
    John Machin, Aug 24, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    354
    Cameron Laird
    Mar 7, 2006
  2. Replies:
    1
    Views:
    250
    Walter Roberson
    Mar 7, 2006
  3. Murali
    Replies:
    2
    Views:
    545
    Jerry Coffin
    Mar 9, 2006
  4. Boris
    Replies:
    0
    Views:
    409
    Boris
    Jul 2, 2008
  5. Replies:
    3
    Views:
    96
    William James
    Mar 7, 2006
Loading...

Share This Page