matching multiple ip addresses

Discussion in 'Perl Misc' started by mhoer@hotmail.com, May 4, 2005.

  1. Guest

    Im fairly new to perl and i have a problem. I am sorting a firewall log
    into a plain text file. I have it doing everything except for 1 thing
    and that is to see a list source and destination IP addresses. I got it
    to print out the data in neat columns.
    The only thing i need is to see if multiple attempts come from or
    are directed to the same ip address. Then the count how many times for
    each mutiple address. If someone could point me in the right direction
    or give me an idea would be great.



    Example of firewall log:
    "Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
    "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
    "Information"
    "1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
    "" ""
    "2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
    "3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""


    ######### Here is what i have so far...

    #!/usr/bin/perl
    # fwlog.plx
    use warnings;
    use strict;


    my $counter = -1;#this is -1 because the first line is not a actual
    line that is needed
    my $numDrops = 0;
    my $numAccepts = 0;
    my $numRejects = 0;
    my @field; #this is for separating the different fields.

    #these are for input and output.
    open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
    open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";

    while(<INPUT>){
    $counter++;#used to find the total
    @field = split /" "/, $_;#this breaks up the file into fields from the
    spaces

    print OUTPUT $field[5],"\t\t";#outputs the origin or name
    print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
    source
    print OUTPUT $field[10],"\n";#outputs the destination

    if ($field[7] =~ /Drop/) {
    $numDrops++;#finds how many drops there are.
    }
    if ($field[7] =~ /Accept/) {
    $numAccepts++;#finds how many accepts there are.
    }
    if ($field[7] =~ /Reject/){
    $numRejects++;#finds how many rejects there are
    }
    }
    # The total of logs, drops, acceps,and rejects
    print OUTPUT "\n\nTotal: ",$counter,"\n\n";
    print OUTPUT "Action:\n";
    print OUTPUT "\tDrops (",$numDrops,")\n";
    print OUTPUT "\tAccepts (",$numAccepts,")\n";
    print OUTPUT "\tRejects (",$numRejects,")\n";

    #still needs to find same sources and destinations and tell how many
    there are.
    , May 4, 2005
    #1
    1. Advertising

  2. wrote:

    > Subject: matching multiple ip addresses


    This is a Perl newgroup. Your subject line should describe what you are
    doing from the point of view of Perl, not from the perspective of your
    application domain

    > Im fairly new to perl and i have a problem. I am sorting a firewall log
    > into a plain text file. I have it doing everything except for 1 thing
    > and that is to see a list source and destination IP addresses. I got it
    > to print out the data in neat columns.
    > The only thing i need is to see if multiple attempts come from or
    > are directed to the same ip address. Then the count how many times for
    > each mutiple address. If someone could point me in the right direction
    > or give me an idea would be great.


    The way to count the instances of distinct values of something is:

    $counter{$something}++;

    > Example of firewall log:
    > "Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
    > "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
    > "Information"
    > "1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
    > "" ""
    > "2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
    > "3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""
    >
    >
    > ######### Here is what i have so far...
    >
    > #!/usr/bin/perl
    > # fwlog.plx
    > use warnings;
    > use strict;


    Excelent start. Well done, you are in the top 20% of first-timers already.

    > my $counter = -1;#this is -1 because the first line is not a actual


    I would actually be temped to initialize counter to 0 read the first
    line from <INPUT> before the loop.

    > my $numDrops = 0;
    > my $numAccepts = 0;
    > my $numRejects = 0;


    This may be OK but it smells to me like you should really be using a
    hash $num{Drops}, $num{Accepts}, $num{Rejects}

    my %num;

    Or if you really want to initialize maybe

    my %num = (Drops => 0, Accepts => 0, Rejects => 0 );

    > my @field; #this is for separating the different fields.


    I think you are suffering from premature declaration. You should always
    declare all variables in the smallest applicable scope. (Note this is
    not Perl specific, it applies to programming in general).

    > #these are for input and output.
    > open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
    > open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";


    Excellent.

    And to throw away the first line...

    <INPUT>;

    > while(<INPUT>){
    > $counter++;#used to find the total
    > @field = split /" "/, $_;#this breaks up the file into fields from the
    > spaces


    There are two ways to split. split() and m//g. The former is useful if
    you have a pattern that describes what you want to skip, the latter if
    you have a pattern that describes what you want to keep.

    my @fields = m/"(.*?)"/g;

    > print OUTPUT $field[5],"\t\t";#outputs the origin or name
    > print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
    > source
    > print OUTPUT $field[10],"\n";#outputs the destination


    Consider using printf.

    printf OUTPUT "%s\t\t%-40s%s", @field[5,9,10];

    >
    > if ($field[7] =~ /Drop/) {
    > $numDrops++;#finds how many drops there are.
    > }
    > if ($field[7] =~ /Accept/) {
    > $numAccepts++;#finds how many accepts there are.
    > }
    > if ($field[7] =~ /Reject/){
    > $numRejects++;#finds how many rejects there are
    > }
    > }


    If you'd used %num hash ...

    $num{$field[7]}++;

    > print OUTPUT "\n\nTotal: ",$counter,"\n\n";
    > print OUTPUT "Action:\n";
    > print OUTPUT "\tDrops (",$numDrops,")\n";
    > print OUTPUT "\tAccepts (",$numAccepts,")\n";
    > print OUTPUT "\tRejects (",$numRejects,")\n";


    IMHO interpolation looks tidier.

    print OUTPUT "\n\nTotal: $counter\n\n";
    print OUTPUT "Action:\n";
    print OUTPUT "\tDrops ($num{Drops})\n";
    print OUTPUT "\tAccepts ($num{Accepts})\n";
    print OUTPUT "\tRejects ($num{Rejects})\n";
    Brian McCauley, May 4, 2005
    #2
    1. Advertising

  3. wrote:
    > Im fairly new to perl and i have a problem. I am sorting a firewall log
    > into a plain text file. I have it doing everything except for 1 thing
    > and that is to see a list source and destination IP addresses. I got it
    > to print out the data in neat columns.
    > The only thing i need is to see if multiple attempts come from or
    > are directed to the same ip address. Then the count how many times for
    > each mutiple address. If someone could point me in the right direction
    > or give me an idea would be great.
    >
    >
    > Example of firewall log:
    > "Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
    > "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
    > "Information"
    > "1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
    > "" ""
    > "2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
    > "3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""
    >
    >
    > ######### Here is what i have so far...
    >
    > #!/usr/bin/perl
    > # fwlog.plx
    > use warnings;
    > use strict;
    >
    >
    > my $counter = -1;#this is -1 because the first line is not a actual
    > line that is needed
    > my $numDrops = 0;
    > my $numAccepts = 0;
    > my $numRejects = 0;
    > my @field; #this is for separating the different fields.
    >
    > #these are for input and output.
    > open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
    > open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";
    >
    > while(<INPUT>){
    > $counter++;#used to find the total
    > @field = split /" "/, $_;#this breaks up the file into fields from the
    > spaces
    >
    > print OUTPUT $field[5],"\t\t";#outputs the origin or name
    > print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
    > source
    > print OUTPUT $field[10],"\n";#outputs the destination
    >
    > if ($field[7] =~ /Drop/) {
    > $numDrops++;#finds how many drops there are.
    > }
    > if ($field[7] =~ /Accept/) {
    > $numAccepts++;#finds how many accepts there are.
    > }
    > if ($field[7] =~ /Reject/){
    > $numRejects++;#finds how many rejects there are
    > }
    > }
    > # The total of logs, drops, acceps,and rejects
    > print OUTPUT "\n\nTotal: ",$counter,"\n\n";
    > print OUTPUT "Action:\n";
    > print OUTPUT "\tDrops (",$numDrops,")\n";
    > print OUTPUT "\tAccepts (",$numAccepts,")\n";
    > print OUTPUT "\tRejects (",$numRejects,")\n";
    >
    > #still needs to find same sources and destinations and tell how many
    > there are.


    You could write it something like this (untested):


    #!/usr/bin/perl
    # fwlog.plx
    use warnings;
    use strict;


    #these are for input and output.
    open INPUT, '<', 'firewalllog2.txt' or die "Couldn't open file: $!\n";
    open OUTPUT, '>', 'fireout.txt' or die "Couldn't open file: $!\n";

    my ( %action, %source, %destination );
    while ( <INPUT> ) {
    #this breaks up the file into fields from the spaces
    my @field = split /" "/;

    #outputs the origin or name, source and destination
    printf OUTPUT "%s\t\t%-40s%s\n", @field[ 5, 9, 10 ];

    $action{ $field[ 7 ] }++;
    $source{ $field[ 9 ] }++;
    $destination{ $field[ 10 ] }++;
    }
    # The total of logs, drops, accepts,and rejects
    printf OUTPUT <<FORMAT, $. - 1, @action{ qw/ Drop Accept Reject / };

    Total: %d

    Action:
    Drops (%d)
    Accepts (%d)
    Rejects (%d)
    FORMAT

    print "\nSource IP Count\n";
    printf "%-15s %d\n", $_, $source{ $_ } for keys %source;
    print "\nDestination IP Count\n";
    printf "%-15s %d\n", $_, $destination{ $_ } for keys %destination;

    __END__



    John
    --
    use Perl;
    program
    fulfillment
    John W. Krahn, May 4, 2005
    #3
  4. Anno Siegel Guest

    <> wrote in comp.lang.perl.misc:
    > Im fairly new to perl and i have a problem. I am sorting a firewall log
    > into a plain text file. I have it doing everything except for 1 thing
    > and that is to see a list source and destination IP addresses. I got it
    > to print out the data in neat columns.
    > The only thing i need is to see if multiple attempts come from or
    > are directed to the same ip address. Then the count how many times for
    > each mutiple address. If someone could point me in the right direction
    > or give me an idea would be great.


    Whenever the question is, "How many times does this or that string
    appear?", the answer in Perl is a hash. Two hashes in your case.

    > Example of firewall log:
    > "Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action"
    > "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User"
    > "Information"
    > "1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719"
    > "" ""
    > "2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
    > "3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW"
    > "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""


    Your news software has broken up the lines. It's a bit of guesswork
    to piece them together again.

    >
    > ######### Here is what i have so far...
    >
    > #!/usr/bin/perl
    > # fwlog.plx
    > use warnings;
    > use strict;
    >
    >
    > my $counter = -1;#this is -1 because the first line is not a actual
    > line that is needed


    Again, line too long, but this time you are in control. Best not to
    post lines longer than 72 characters to Usenet.

    Further, if you are going to throw the first line away, do so. Don't
    rely on the fact that the rest of the program will ignore it. That
    may change some day.

    However, the header line you got there is pure gold. Giving names
    to the data fields, it allows you to write your program in a more
    self-documenting manner. See my rewrite of your program below for
    how to do that.

    > my $numDrops = 0;
    > my $numAccepts = 0;
    > my $numRejects = 0;
    > my @field; #this is for separating the different fields.
    >
    > #these are for input and output.


    Needless comment. You *named* them INPUT and OUTPUT.

    > open INPUT, "firewalllog2.txt" or die "Couldn't open file: $!\n";
    > open OUTPUT, ">fireout.txt" or die "Couldn't open file: $!\n";


    You should mention the file name in the error message so the user knows
    which file could not be opened.

    >
    > while(<INPUT>){
    > $counter++;#used to find the total
    > @field = split /" "/, $_;#this breaks up the file into fields from the
    > spaces


    The simple split may not be good enough. The log format looks a lot like
    it has some escaping mechanism up its sleeve if a string should contain
    quote marks. If so, Text::Balanced or Text::parsewords may be the
    answer.

    > print OUTPUT $field[5],"\t\t";#outputs the origin or name
    > print OUTPUT $field[9]," " x (40-length($field[9]));#outputs the
    > source
    > print OUTPUT $field[10],"\n";#outputs the destination
    >
    > if ($field[7] =~ /Drop/) {
    > $numDrops++;#finds how many drops there are.
    > }
    > if ($field[7] =~ /Accept/) {
    > $numAccepts++;#finds how many accepts there are.
    > }
    > if ($field[7] =~ /Reject/){
    > $numRejects++;#finds how many rejects there are
    > }
    > }
    > # The total of logs, drops, acceps,and rejects
    > print OUTPUT "\n\nTotal: ",$counter,"\n\n";
    > print OUTPUT "Action:\n";
    > print OUTPUT "\tDrops (",$numDrops,")\n";
    > print OUTPUT "\tAccepts (",$numAccepts,")\n";
    > print OUTPUT "\tRejects (",$numRejects,")\n";
    >
    > #still needs to find same sources and destinations and tell how many
    > there are.


    I have adapted your program below to do this.

    To count how often a destination $dest appears, use a hash %destcount
    and increment the value $destcount{ $dest} as each comes by. Same for
    the sources. Since there are two such hashes to deal with, I have
    used a subroutine to report their content.

    Another change, as mentioned, is the use of the header line to collect
    possible hash keys in @keys. For that purpose, your array @fields has
    been replaced by a hash %field. It has the same function, but instead
    of accessing fields through magic numbers, it accesses them through
    meaningful strings.

    open OUTPUT, '>&', 'STDOUT' or die "Can't dup STDOUT"; # for simplicity

    my @keys;
    @keys = /"([^"]+)"/g for scalar <DATA>;

    my $counter = 0;
    my $numDrops = 0;
    my $numAccepts = 0;
    my $numRejects = 0;
    my ( %field, %sourcecount, %destcount);
    while(<DATA>){
    ++ $counter;
    @field{ @keys} = /"([^"]+)"/g;

    print "$field{ Origin}\t\t";
    print $field{ Source}, " " x (40-length( $field{ Source}));
    print $field{ Destination},"\n";
    ++ $destcount{ $field{ Destination}};
    ++ $sourcecount{ $field{ Source}};

    for ( $field{ Action} ) {
    ++ $numDrops if /Drop/;
    ++ $numAccepts if /Accept/;
    ++ $numRejects if /Reject/;
    }
    }
    print "\n\nTotal: ",$counter,"\n\n";
    print "Action:\n";
    print "\tDrops (",$numDrops,")\n";
    print "\tAccepts (",$numAccepts,")\n";
    print "\tRejects (",$numRejects,")\n";

    report_count( "N of attempts by source", \ %sourcecount);
    report_count( "N of attempts by destination", \ %destcount);

    sub report_count {
    my ( $msg, $count) = @_;
    my $any;
    print "$msg:\n";
    for ( sort keys %$count ) {
    next if $count->{ $_} < 2;
    ++ $any;
    print "$_: $count->{ $_}\n";
    }
    print "No multiple attempts\n" unless $any;
    }

    __DATA__
    "Number" "Date" "Time" "Product" "Interface" "Origin" "Type" "Action" "Service" "Source" "Destination" "Protocol" "Rule" "Source Port" "User" "Information"
    "1" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719" "" ""
    "2" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Accept" "imap" "DT017484" "emh1.otc.edu" "tcp" "30" "4036" "" ""
    "3" "19Sep2003" "17:00:05" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "389" "172.16.16.179" "192.0.0.8" "udp" "92" "1068" "" ""
    "4" "19Sep2003" "17:00:01" "VPN-1 & FireWall-1" "N100016" "OTC-FW" "Log" "Drop" "smtp" "192.168.3.98" "64.12.137.152" "tcp" "92" "35719" "" ""
    Anno Siegel, May 4, 2005
    #4
  5. Brian McCauley <> wrote in
    news:d5b63o$d6k$:

    > wrote:
    >

    ....

    >> #!/usr/bin/perl
    >> # fwlog.plx
    >> use warnings;
    >> use strict;

    >
    > Excelent start. Well done, you are in the top 20% of first-timers
    > already.


    Ditto. Thanks to the OP.

    ....

    > Excellent.
    >
    > And to throw away the first line...
    >
    > <INPUT>;
    >
    >> while(<INPUT>){
    >> $counter++;#used to find the total


    It seems to me $counter only really counts the total lines. In that
    case, can't the whole thing be reduced to:
    #! /usr/bin/perl

    use strict;
    use warnings;

    while(<DATA>) {
    next unless $.;
    print;
    }

    print "$. lines read";

    __END__
    1
    2
    3
    4

    Or, would it better to localize $. before the while loop?

    {
    local $.;
    while(<DATA>) {
    next unless $.;
    print;
    }
    print "$. lines read";
    }

    Sinan

    --
    A. Sinan Unur <>
    (reverse each component and remove .invalid for email address)

    comp.lang.perl.misc guidelines on the WWW:
    http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html
    A. Sinan Unur, May 5, 2005
    #5
  6. <> wrote in message
    news:...
    > Im fairly new to perl and i have a problem. I am sorting a firewall log
    > into a plain text file. I have it doing everything except for 1 thing
    > and that is to see a list source and destination IP addresses. I got it
    > to print out the data in neat columns.
    > The only thing i need is to see if multiple attempts come from or
    > are directed to the same ip address. Then the count how many times for
    > each mutiple address. If someone could point me in the right direction
    > or give me an idea would be great.

    <> wrote in message
    news:...
    > Im fairly new to perl and i have a problem. I am sorting a firewall log
    > into a plain text file. I have it doing everything except for 1 thing
    > and that is to see a list source and destination IP addresses. I got it
    > to print out the data in neat columns.
    > The only thing i need is to see if multiple attempts come from or
    > are directed to the same ip address. Then the count how many times for
    > each mutiple address. If someone could point me in the right direction
    > or give me an idea would be great.


    Sounds like fwlogum might be useful for you.

    See http://www.ginini.com/software/fwlogsum/

    Please note that you should not use the code from this as an example for
    learning. It was originally written back
    in the days of Perl 4, and is currently being completely rewritten to be
    much better structured.
    Peter Sundstrom, May 5, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andrew McLean

    Fuzzy matching of postal addresses

    Andrew McLean, Jan 18, 2005, in forum: Python
    Replies:
    18
    Views:
    709
    Joseph Turian
    Jan 24, 2005
  2. Tim Churches

    Re: Fuzzy matching of postal addresses

    Tim Churches, Jan 18, 2005, in forum: Python
    Replies:
    0
    Views:
    479
    Tim Churches
    Jan 18, 2005
  3. Tim Churches
    Replies:
    4
    Views:
    516
    Tim Churches
    Feb 20, 2005
  4. namespace1
    Replies:
    3
    Views:
    890
  5. Bobby Chamness
    Replies:
    2
    Views:
    224
    Xicheng Jia
    May 3, 2007
Loading...

Share This Page