Reading last N lines from large file

Discussion in 'Perl Misc' started by Chris Nestrud, Nov 25, 2004.

  1. I need to read a number of lines from the end of a large file. For
    example, the file may have 460484 lines and I need the last 500.

    The "tail -500 file" command from the shell works, but I'd like a solution
    that doesn't involve calling an external program.

    Reading the file and keeping an array with the last 500 lines seen
    works, but takes a very long time on large files.

    I've looked at the seek function, but other than guessing at how far
    from EOF to seek backward, I can't figure out how to make sure that I'm
    left with enough data to account for the needed number of lines.

    Any suggestions or other approaches would be much appreciated.
    --
    Chris Nestrud
    Email:
    http://www.panix.com/~ccn/
    Chris Nestrud, Nov 25, 2004
    #1
    1. Advertising

  2. Chris Nestrud <> wrote:
    [...]
    > I've looked at the seek function, but other than guessing at how far
    > from EOF to seek backward, I can't figure out how to make sure that
    > I'm left with enough data to account for the needed number of lines.


    I suspect tail merely guesses a seek position, then reads to find out
    if it was right, retrying until it gets enough data.


    --
    PGP key ID E85DC776 - finger for full key
    Peter Corlett, Nov 25, 2004
    #2
    1. Advertising

  3. Chris Nestrud

    Tore Aursand Guest

    On Thu, 25 Nov 2004 14:51:19 +0000, Chris Nestrud wrote:
    > I need to read a number of lines from the end of a large file. For
    > example, the file may have 460484 lines and I need the last 500.


    Take a look at the 'File::ReadBackwards' module;

    <http://search.cpan.org/~uri/File-ReadBackwards-1.02/>


    --
    Tore Aursand <>
    "A teacher is never a giver of truth - he is a guide, a pointer to the
    truth that each student must find for himself. A good teacher is
    merely a catalyst." (Bruce Lee)
    Tore Aursand, Nov 25, 2004
    #3
  4. On Thu, 25 Nov 2004 17:01:49 +0100, Tore Aursand <> wrote:
    > On Thu, 25 Nov 2004 14:51:19 +0000, Chris Nestrud wrote:
    >> I need to read a number of lines from the end of a large file. For
    >> example, the file may have 460484 lines and I need the last 500.

    >
    > Take a look at the 'File::ReadBackwards' module;
    >
    > <http://search.cpan.org/~uri/File-ReadBackwards-1.02/>


    This looks like exactly what I need. Thanks.
    --
    Chris Nestrud
    Email:
    http://www.panix.com/~ccn/
    Chris Nestrud, Nov 25, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    10
    Views:
    2,416
  2. Joe Wright
    Replies:
    0
    Views:
    496
    Joe Wright
    Jul 27, 2003
  3. nic977
    Replies:
    14
    Views:
    1,055
    Irrwahn Grausewitz
    Sep 13, 2003
  4. Rui Vilao
    Replies:
    21
    Views:
    275
    Mihai N.
    Dec 9, 2003
  5. PerlFAQ Server
    Replies:
    0
    Views:
    143
    PerlFAQ Server
    Jan 31, 2011
Loading...

Share This Page