(noob) need help parsing Apache log file

Discussion in 'Ruby' started by Koncept, Feb 28, 2004.

  1. Koncept

    Koncept Guest

    Being very new to Ruby, I want to be able to write a small script to
    open my Apache access_log file and remove any lines that contain (\.exe
    | \.ida) from the source file. I managed to figure out how to get this
    information and log it to a new file, but I would really appreciate an
    example of how to read and remove from one file. I am the kind of
    person who learns best watching others.


    This is what I wrote to parse the file. Any revisions, hints, hacks,
    etc would also be very much appreciated.

    src = '/var/log/httpd/access_log'
    arr = IO.readlines(src)
    len = arr.length
    lw = 73
    msEvil = Array.new
    div = '--'

    arr.each do |line|
    msEvil.push line if line =~ /(\.exe| \.ida)/
    end

    # Generate report
    puts ('Report for: ' + Time.new.to_s).center(lw)
    puts (div * 20).center(lw)
    puts
    puts 'Total lines:'.ljust(lw/2) + len.to_s.rjust(lw/2)
    puts 'Total Window exploits:'.ljust(lw/2) +
    msEvil.length.to_s.rjust(lw/2)

    # log to text file
    f = File.new("codeRed.txt", "w+")

    msEvil.each do |line|
    f.puts line
    end

    f.close

    --
    Koncept <<
    "Contrary to popular belief, the most dangerous animal is not the lion or
    tiger or even the elephant. The most dangerous animal is a shark riding
    on an elephant, just trampling and eating everything they see." - Jack Handey
    Koncept, Feb 28, 2004
    #1
    1. Advertising

  2. Koncept

    Mark Hubbart Guest

    On Feb 27, 2004, at 10:19 PM, Koncept wrote:

    > Being very new to Ruby, I want to be able to write a small script to
    > open my Apache access_log file and remove any lines that contain (\.exe
    > | \.ida) from the source file. I managed to figure out how to get this
    > information and log it to a new file, but I would really appreciate an
    > example of how to read and remove from one file. I am the kind of
    > person who learns best watching others.
    >
    >
    > This is what I wrote to parse the file. Any revisions, hints, hacks,
    > etc would also be very much appreciated.


    One small change will write out the file. Instead of:

    > arr.each do |line|
    > msEvil.push line if line =~ /(\.exe| \.ida)/
    > end


    try:

    # reject! deletes matched lines
    arr.reject! do |line|
    msEvil.push line if line =~ /(\.exe| \.ida)/
    end
    # truncate the file and write out
    File.open(src, "w+"){|f| f.write arr.join }


    Array#reject! does the same thing as Array#sort, except it pays
    attention to the return value of the passed block. If the block returns
    a true value, it deletes the item from the array. There is also the
    non-destructive version, Array#reject (no bang), but reject! makes more
    sense here.

    File.open opens the file (surprise!). In this case, it truncates it and
    opens it write only. If you give it a block, it passes the opened file
    handle to the block, and automagically closes the file when it's done
    :) You might want to use the same trick at the bottom:

    > # log to text file
    > f = File.new("codeRed.txt", "w+")
    >
    > msEvil.each do |line|
    > f.puts line
    > end
    >
    > f.close


    could become:

    # log to text file
    File.open("codeRed.txt", "w+"){|f| f.write msEvil.join }


    That's a handy Ruby idiom. Shortens code considerably :)

    And finally, here's the results of running this script on my logs:

    mark@imac% sudo mslogstripminer
    Password:
    Report for: Sat Feb 28 00:06:42 PST 2004
    ----------------------------------------

    Total lines: 1670
    Total Window exploits: 104
    mark@imac%

    And that's my box at home!!! absolutely disgusting. :)

    --Mark
    Mark Hubbart, Feb 28, 2004
    #2
    1. Advertising

  3. Koncept

    Koncept Guest

    In article <>, Mark Hubbart
    <> wrote:

    > mark@imac% sudo mslogstripminer
    > Password:
    > Report for: Sat Feb 28 00:06:42 PST 2004
    > ----------------------------------------
    >
    > Total lines: 1670
    > Total Window exploits: 104
    > mark@imac%
    >
    > And that's my box at home!!! absolutely disgusting. :)


    You think you have it bad... This is MY home box.

    Report for: Sat Feb 28 03:53:05 EST 2004

    ----------------------------------------


    Total lines: 31068
    Total Window exploits: 637

    Thanks for you help and tips. Everything made total sense and you gave
    me a few pointers which will save tons of time. Appreciate your help.

    Cheers

    --
    Koncept <<
    "Contrary to popular belief, the most dangerous animal is not the lion or
    tiger or even the elephant. The most dangerous animal is a shark riding
    on an elephant, just trampling and eating everything they see." - Jack Handey
    Koncept, Feb 28, 2004
    #3
  4. Koncept

    Koncept Guest

    Mark. One other thing....

    If the script is run without *sudo* permissions, this is the error
    thrown. How do I catch something like this and present a nicer
    message to the user?

    logReader.rb:29:in `initialize': Permission denied -
    /var/log/httpd/access_log (Errno::EACCES)
    from logReader.rb:29:in `open'
    from logReader.rb:29

    --
    Koncept <<
    "Contrary to popular belief, the most dangerous animal is not the lion or
    tiger or even the elephant. The most dangerous animal is a shark riding
    on an elephant, just trampling and eating everything they see." - Jack Handey
    Koncept, Feb 28, 2004
    #4
  5. Koncept

    Mark Hubbart Guest

    On Feb 28, 2004, at 10:24 AM, Koncept wrote:

    > Mark. One other thing....
    >
    > If the script is run without *sudo* permissions, this is the error
    > thrown. How do I catch something like this and present a nicer
    > message to the user?


    You need to catch the errors. Enclose the error-generating code in a
    begin..rescue..end statement:

    begin
    # error-prone code here
    rescue
    warn "You don't have permission to access the log. Try using sudo."
    exit
    end

    --

    Just replace the comment with the line (or lines) of code. You'll
    probably want to do this with the code where you read in the file, and
    also the two places where you write the data out. For much more
    detailed information on error catching, refer to the PickAxe Guide[1],
    "Exceptions, Catch, and Throw".

    --Mark

    [1]: http://phrogz.net/ProgrammingRuby/
    Mark Hubbart, Feb 28, 2004
    #5
  6. Koncept

    Koncept Guest

    In article <>, Mark Hubbart
    <> wrote:

    > Just replace the comment with the line (or lines) of code. You'll
    > probably want to do this with the code where you read in the file, and
    > also the two places where you write the data out. For much more
    > detailed information on error catching, refer to the PickAxe Guide[1],
    > "Exceptions, Catch, and Throw".


    Mark. Thanks for all you help.

    It is so nice to be a part of a group ( unlike those Perl folks ) where
    people are actually nice to each other.

    you = "the man"

    --
    Koncept <<
    "Contrary to popular belief, the most dangerous animal is not the lion or
    tiger or even the elephant. The most dangerous animal is a shark riding
    on an elephant, just trampling and eating everything they see." - Jack Handey
    Koncept, Feb 28, 2004
    #6
  7. Koncept

    Mark Hubbart Guest

    On Feb 28, 2004, at 1:19 PM, Koncept wrote:

    > It is so nice to be a part of a group ( unlike those Perl folks ) where
    > people are actually nice to each other.


    We're all nice here because programming in ruby is fun :) as for
    Perl... I'd better not say anything.

    --Mark
    Mark Hubbart, Feb 28, 2004
    #7
  8. Mark Hubbart <> writes:
    > On Feb 28, 2004, at 1:19 PM, Koncept wrote:
    >> It is so nice to be a part of a group ( unlike those Perl folks ) where
    >> people are actually nice to each other.

    >
    > We're all nice here because programming in ruby is fun :) as for
    > Perl... I'd better not say anything.


    To be fair, unlike Ruby, Perl has *tons* of documentation distributed
    with it that a poster could easily consult before posting a question;
    where it's not fair to tell someone to just "Read the Pickaxe", since
    that's not definitive or complete, it is fairly reasonable for a Perl
    programmer to be told to read the standard documentation on her hard
    drive before asking a question it could easily answer.

    Hopefully, the hard work being put into the RDoc<->ri integration
    effort will result in similarly high-quality documentation being
    included with a default distribution of Ruby.

    -=Eric
    --
    Come to think of it, there are already a million monkeys on a million
    typewriters, and Usenet is NOTHING like Shakespeare.
    -- Blair Houghton.
    Eric Schwartz, Mar 3, 2004
    #8
  9. Koncept

    Mac Guest

    Eric Schwartz wrote:
    > Mark Hubbart <> writes:
    >
    >>On Feb 28, 2004, at 1:19 PM, Koncept wrote:
    >>
    >>>It is so nice to be a part of a group ( unlike those Perl folks ) where
    >>>people are actually nice to each other.

    >>
    >>We're all nice here because programming in ruby is fun :) as for
    >>Perl... I'd better not say anything.

    >
    >
    > To be fair, unlike Ruby, Perl has *tons* of documentation distributed
    > with it that a poster could easily consult before posting a question;
    > where it's not fair to tell someone to just "Read the Pickaxe", since
    > that's not definitive or complete, it is fairly reasonable for a Perl
    > programmer to be told to read the standard documentation on her hard
    > drive before asking a question it could easily answer.
    >
    > Hopefully, the hard work being put into the RDoc<->ri integration
    > effort will result in similarly high-quality documentation being
    > included with a default distribution of Ruby.
    >
    > -=Eric


    To be realy fair, as a perl newbie (and even now) I found/find the Perl
    docs to be unstructured, disjointed, scattered within the bundle. It
    reflects the history of the language, not a concise view of how the
    language works and how to get things done.

    It also reflects (im very ho) the sometimes hard choices that were made
    to keep the language alive in the face of the requirements of it's users
    without completely breaking with it's history.

    As to reponding to us newbies/nubys, Elwood P. Dowd (the movie "Harvey")
    said he recommended " pleasant". Even in the face of documentation.
    Hopefully you're not suffering fools, you're helping people on an
    intellectual journey.

    Mac
    Mac, Mar 3, 2004
    #9
  10. Koncept

    Mark Hubbart Guest

    On Mar 2, 2004, at 4:49 PM, Eric Schwartz wrote:

    > Mark Hubbart <> writes:
    >> On Feb 28, 2004, at 1:19 PM, Koncept wrote:
    >>> It is so nice to be a part of a group ( unlike those Perl folks )
    >>> where
    >>> people are actually nice to each other.

    >>
    >> We're all nice here because programming in ruby is fun :) as for
    >> Perl... I'd better not say anything.

    >
    > To be fair, unlike Ruby, Perl has *tons* of documentation distributed
    > with it that a poster could easily consult before posting a question;
    > where it's not fair to tell someone to just "Read the Pickaxe", since
    > that's not definitive or complete, it is fairly reasonable for a Perl
    > programmer to be told to read the standard documentation on her hard
    > drive before asking a question it could easily answer.
    >
    > Hopefully, the hard work being put into the RDoc<->ri integration
    > effort will result in similarly high-quality documentation being
    > included with a default distribution of Ruby.


    what I meant was:
    We're all nice here because programming in ruby is fun. As for
    Perl... I'd better not say anything. :)

    I think that's clearer :)

    Regardless, even if Ruby WAS superbly documented, I still think newbies
    should be responded to in the same way: a quick example, and a
    reference to more information. When you are new to a programming
    language, sometimes you only know what the result of your code *should*
    be, and have no way of knowing how to go about actually *getting* it.
    So even if you did look in the docs, you might not know what to look
    for.

    Simple questions from these "extreme newbies", IMHO, should be left for
    the more advanced newbies. If they were treated the same way, they will
    probably be happy to answer them, sort of giving back to the group.

    So everyone should feel it's okay to jump in; ask a question, give a
    response if you think it's correct, correct someone else if they are
    wrong, show another way of doing it. Politely. With this kind of open
    communication, people can say things without fear of getting
    backhanded. Good temperaments in a community can be a big draw, and it
    can be easily evidenced by interactions in the newsgroup :)

    heh. that sounded really cheezy, but you know what I mean.

    Just my 2ยข

    --Mark
    Mark Hubbart, Mar 3, 2004
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Henrik_the_boss
    Replies:
    0
    Views:
    2,643
    Henrik_the_boss
    Nov 5, 2003
  2. Jim Richardson

    Parsing apache log files

    Jim Richardson, Feb 20, 2004, in forum: Python
    Replies:
    9
    Views:
    732
    Jim Richardson
    Feb 22, 2004
  3. AAaron123
    Replies:
    1
    Views:
    274
    Cowboy \(Gregory A. Beamer\)
    Oct 14, 2008
  4. Joe Nciri

    Parsing an apache access log line

    Joe Nciri, Jul 16, 2007, in forum: Ruby
    Replies:
    5
    Views:
    109
    Phil Meier
    Jul 16, 2007
  5. ad
    Replies:
    3
    Views:
    203
    Ryan Davis
    Dec 29, 2009
Loading...

Share This Page