Split a very large log file

Discussion in 'Perl Misc' started by Terry.Riegel@gmail.com, Feb 2, 2007.

  1. Guest

    I have a 9gig log file, I am trying to do something with it. I tried
    to run it through awstats and after about 8 hours I got a segmentation
    fault.

    I think if I could break it up into 2 files I could process one , then
    process the other. I am not very good at PERL, but have to believe
    this would be a 5-10 line program.

    Any ideas?

    Thanks,

    Terry Riegel
    , Feb 2, 2007
    #1
    1. Advertising

  2. J. Gleixner Guest

    wrote:
    > I have a 9gig log file, I am trying to do something with it. I tried
    > to run it through awstats and after about 8 hours I got a segmentation
    > fault.
    >
    > I think if I could break it up into 2 files I could process one , then
    > process the other. I am not very good at PERL, but have to believe
    > this would be a 5-10 line program.


    Yep.. why not post what you've tried.

    Also, if you're on *nix? There are a bunch of
    commands that will do it:

    man split
    man head
    man tail
    J. Gleixner, Feb 2, 2007
    #2
    1. Advertising

  3. Guest

    wrote:
    > I have a 9gig log file, I am trying to do something with it. I tried
    > to run it through awstats and after about 8 hours I got a segmentation
    > fault.
    >
    > I think if I could break it up into 2 files I could process one , then
    > process the other.


    I would guess that doing this will lead to the wrong answers. If awstats
    is at all well written, then it wouldn't be loading all the data into
    memory (which I'm guessing is what precipitated the seg fault) unless that
    is what it needed to do to get the right answer.


    > I am not very good at PERL, but have to believe
    > this would be a 5-10 line program.


    I'd use GNU "split" rather than using Perl at all, but anyway you just
    need to look in $. to decide if you should reopen to a new output
    file.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Feb 3, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Raymond Arthur St. Marie II of III

    very Very VERY dumb Question About The new Set( ) 's

    Raymond Arthur St. Marie II of III, Jul 23, 2003, in forum: Python
    Replies:
    4
    Views:
    453
    Raymond Hettinger
    Jul 27, 2003
  2. shanx__=|;-

    very very very long integer

    shanx__=|;-, Oct 16, 2004, in forum: C Programming
    Replies:
    19
    Views:
    1,594
    Merrill & Michele
    Oct 19, 2004
  3. Abhishek Jha

    very very very long integer

    Abhishek Jha, Oct 16, 2004, in forum: C Programming
    Replies:
    4
    Views:
    410
    jacob navia
    Oct 17, 2004
  4. Peter

    Very very very basic question

    Peter, Feb 8, 2005, in forum: C Programming
    Replies:
    14
    Views:
    503
    Dave Thompson
    Feb 14, 2005
  5. Curious Joe
    Replies:
    2
    Views:
    366
Loading...

Share This Page