(noob) need help parsing Apache log file

[Koncept]

Being very new to Ruby, I want to be able to write a small script to
open my Apache access_log file and remove any lines that contain (\.exe
| \.ida) from the source file. I managed to figure out how to get this
information and log it to a new file, but I would really appreciate an
example of how to read and remove from one file. I am the kind of
person who learns best watching others.


This is what I wrote to parse the file. Any revisions, hints, hacks,
etc would also be very much appreciated.

src = '/var/log/httpd/access_log'
arr = IO.readlines(src)
len = arr.length
lw = 73
msEvil = Array.new
div = '--'

arr.each do |line|
msEvil.push line if line =~ /(\.exe| \.ida)/
end

# Generate report
puts ('Report for: ' + Time.new.to_s).center(lw)
puts (div * 20).center(lw)
puts
puts 'Total lines:'.ljust(lw/2) + len.to_s.rjust(lw/2)
puts 'Total Window exploits:'.ljust(lw/2) +
msEvil.length.to_s.rjust(lw/2)

# log to text file
f = File.new("codeRed.txt", "w+")

msEvil.each do |line|
f.puts line
end

f.close

 

[Mark Hubbart]

Being very new to Ruby, I want to be able to write a small script to
open my Apache access_log file and remove any lines that contain (\.exe
| \.ida) from the source file. I managed to figure out how to get this
information and log it to a new file, but I would really appreciate an
example of how to read and remove from one file. I am the kind of
person who learns best watching others.


This is what I wrote to parse the file. Any revisions, hints, hacks,
etc would also be very much appreciated.

One small change will write out the file. Instead of:

arr.each do |line|
msEvil.push line if line =~ /(\.exe| \.ida)/
end

try:

# reject! deletes matched lines
arr.reject! do |line|
msEvil.push line if line =~ /(\.exe| \.ida)/
end
# truncate the file and write out
File.open(src, "w+"){|f| f.write arr.join }


Array#reject! does the same thing as Array#sort, except it pays
attention to the return value of the passed block. If the block returns
a true value, it deletes the item from the array. There is also the
non-destructive version, Array#reject (no bang), but reject! makes more
sense here.

File.open opens the file (surprise!). In this case, it truncates it and
opens it write only. If you give it a block, it passes the opened file
handle to the block, and automagically closes the file when it's done
You might want to use the same trick at the bottom:

# log to text file
f = File.new("codeRed.txt", "w+")

msEvil.each do |line|
f.puts line
end

f.close

could become:

# log to text file
File.open("codeRed.txt", "w+"){|f| f.write msEvil.join }


That's a handy Ruby idiom. Shortens code considerably

And finally, here's the results of running this script on my logs:

mark@imac% sudo mslogstripminer
Password:
Report for: Sat Feb 28 00:06:42 PST 2004
----------------------------------------

Total lines: 1670
Total Window exploits: 104
mark@imac%

And that's my box at home!!! absolutely disgusting.

--Mark

 

[Koncept]

mark@imac% sudo mslogstripminer
Password:
Report for: Sat Feb 28 00:06:42 PST 2004
----------------------------------------

Total lines: 1670
Total Window exploits: 104
mark@imac%

And that's my box at home!!! absolutely disgusting.

You think you have it bad... This is MY home box.

Report for: Sat Feb 28 03:53:05 EST 2004

----------------------------------------


Total lines: 31068
Total Window exploits: 637

Thanks for you help and tips. Everything made total sense and you gave
me a few pointers which will save tons of time. Appreciate your help.

Cheers

 

[Koncept]

Mark. One other thing....

If the script is run without *sudo* permissions, this is the error
thrown. How do I catch something like this and present a nicer
message to the user?

logReader.rb:29:in `initialize': Permission denied -
/var/log/httpd/access_log (Errno::EACCES)
from logReader.rb:29:in `open'
from logReader.rb:29

 

[Mark Hubbart]

Mark. One other thing....

If the script is run without *sudo* permissions, this is the error
thrown. How do I catch something like this and present a nicer
message to the user?

You need to catch the errors. Enclose the error-generating code in a
begin..rescue..end statement:

begin
# error-prone code here
rescue
warn "You don't have permission to access the log. Try using sudo."
exit
end

--

Just replace the comment with the line (or lines) of code. You'll
probably want to do this with the code where you read in the file, and
also the two places where you write the data out. For much more
detailed information on error catching, refer to the PickAxe Guide[1],
"Exceptions, Catch, and Throw".

--Mark

[1]: http://phrogz.net/ProgrammingRuby/

 

[Koncept]

Just replace the comment with the line (or lines) of code. You'll
probably want to do this with the code where you read in the file, and
also the two places where you write the data out. For much more
detailed information on error catching, refer to the PickAxe Guide[1],
"Exceptions, Catch, and Throw".

Mark. Thanks for all you help.

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.

you = "the man"

 

[Mark Hubbart]

It is so nice to be a part of a group ( unlike those Perl folks ) where
people are actually nice to each other.

We're all nice here because programming in ruby is fun as for
Perl... I'd better not say anything.

--Mark

 

[Eric Schwartz]

We're all nice here because programming in ruby is fun as for
Perl... I'd better not say anything.

To be fair, unlike Ruby, Perl has *tons* of documentation distributed
with it that a poster could easily consult before posting a question;
where it's not fair to tell someone to just "Read the Pickaxe", since
that's not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

-=Eric

 

[Mac]

To be fair, unlike Ruby, Perl has *tons* of documentation distributed
with it that a poster could easily consult before posting a question;
where it's not fair to tell someone to just "Read the Pickaxe", since
that's not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

-=Eric

To be realy fair, as a perl newbie (and even now) I found/find the Perl
docs to be unstructured, disjointed, scattered within the bundle. It
reflects the history of the language, not a concise view of how the
language works and how to get things done.

It also reflects (im very ho) the sometimes hard choices that were made
to keep the language alive in the face of the requirements of it's users
without completely breaking with it's history.

As to reponding to us newbies/nubys, Elwood P. Dowd (the movie "Harvey")
said he recommended " pleasant". Even in the face of documentation.
Hopefully you're not suffering fools, you're helping people on an
intellectual journey.

Mac

 

[Mark Hubbart]

To be fair, unlike Ruby, Perl has *tons* of documentation distributed
with it that a poster could easily consult before posting a question;
where it's not fair to tell someone to just "Read the Pickaxe", since
that's not definitive or complete, it is fairly reasonable for a Perl
programmer to be told to read the standard documentation on her hard
drive before asking a question it could easily answer.

Hopefully, the hard work being put into the RDoc<->ri integration
effort will result in similarly high-quality documentation being
included with a default distribution of Ruby.

what I meant was:
We're all nice here because programming in ruby is fun. As for
Perl... I'd better not say anything.

I think that's clearer

Regardless, even if Ruby WAS superbly documented, I still think newbies
should be responded to in the same way: a quick example, and a
reference to more information. When you are new to a programming
language, sometimes you only know what the result of your code *should*
be, and have no way of knowing how to go about actually *getting* it.
So even if you did look in the docs, you might not know what to look
for.

Simple questions from these "extreme newbies", IMHO, should be left for
the more advanced newbies. If they were treated the same way, they will
probably be happy to answer them, sort of giving back to the group.

So everyone should feel it's okay to jump in; ask a question, give a
response if you think it's correct, correct someone else if they are
wrong, show another way of doing it. Politely. With this kind of open
communication, people can say things without fear of getting
backhanded. Good temperaments in a community can be a big draw, and it
can be easily evidenced by interactions in the newsgroup

heh. that sounded really cheezy, but you know what I mean.

Just my 2¢

--Mark