parsing log in multiple passes

Discussion in 'Perl Misc' started by disco, Aug 4, 2003.

  1. disco

    disco Guest

    I have output from a CLI that I am parsing into a data structure
    (hash) to be used as input to another program which reads the hash.
    The CLI always formats the data in the same by first listing the card
    information followed by information for each port.

    I have figured out the routine to parse all of the data into a hash,
    but what I can't figure out is how to do determine the unique key
    value first by considering all of the CLI output and then use it as I
    build the hash. There is probably a better way to avoid rewinding the
    file as I've done too.

    Right now I just use the Node Id, but I really need my unique key to
    be based on the following criteria:

    1) If the "node id" <> 0's then use the Node ID as the unique
    identifier.
    2) If the "node id" == 0's then use the port id with the smallest
    alpha value (it is in hex) as the unique identifier.

    <cli.log>

    Card name : SuperWombat
    Firmware Version : 4.234
    Driver Version : 7.2345
    Node Id : 00000
    Number of Ports : 2

    Port Id : 1a3c5f
    Port name : Alpha
    Port type : Ethernet
    Port speed : 10

    Port Id : af2c2d2
    Port name : Alpha2
    Port type : Scsi
    Port speed : 1gig

    Card name : BitBlaster
    Firmware Version : 5.000
    Driver Version : 8.000 rel 5.02
    Node Id : 24550
    Number of Ports : 1

    Port Id : 10200
    Port name : GigE1
    Port type : Ethernet
    Port speed : 1000

    Ideal output (understanding that in real life the hash doesn't come
    out in order -- that doesn't matter, and that "->" is simply for
    presentation and not part of data)

    1a3c5f -> Card name -> SuperWombat
    1a3c5f -> Firmware Version -> 4.234
    1a3c5f -> Driver Version -> 7.2345
    1a3c5f -> Node Id -> 00000
    1a3c5f -> Number of Ports -> 2
    1a3c5f -> 0 -> Port Id -> 1a3c5f
    1a3c5f -> 0 -> Port name -> Alpha
    1a3c5f -> 0 -> Port type -> Ethernet
    1a3c5f -> 0 -> Port speed -> 10
    1a3c5f -> 1 -> Port Id -> af2c2d2
    1a3c5f -> 1 -> Port name -> Alpha2
    1a3c5f -> 1 -> Port type -> Scsi
    1a3c5f -> 1 -> Port speed -> 1gig
    24550 -> Card name -> BitBlaster
    24550 -> Firmware Version -> 5.000
    24550 -> Driver Version -> 8.000 rel 5.02
    24550 -> Node Id -> 24550
    24550 -> Number of Ports -> 1
    24550 -> 0 -> Port Id -> 10200
    24550 -> 0 -> Port name -> GigE1
    24550 -> 0 -> Port type -> Ethernet
    24550 -> 0 -> Port speed -> 1000


    Can anyone suggest a way to do this easily? I have spent alot of time
    trying to push/pop/sort arrays, but I can't pull the whole thing
    together. I know this can be done in Perl, but I am stuck.

    Thanks!
    Disco

    following is my code that shows what I have done so far...

    #!/usr/bin/perl
    use strict;

    my $line; # Scalar to hold line from log file being processed.
    my $card_hash = {}; #anonymous hash (data structure)
    my $card_key; # Primary key
    my $tkey; # Sub key
    my $tval; # Value

    my $port_count = 0;

    my @temp;
    my @card_tag;

    open (DATA, "cli.log")or die("Cannot open: filename $!");

    # Find node_wwn's to use as a unique identifier for each card.
    # Create an array of node WWN's found and then rewind the file and
    start again.
    #

    while (<DATA>)
    {
    chomp;
    push @temp, $_ if /Node Id/; # get the entire line if it
    contains "Node Id" and add it to array
    }

    foreach $line (@temp) # assign each item in list (array) to $line and
    process
    {
    $line =~ /(.*?):(.*)/; # Delimiter ==> " : "
    $tval = $2;
    $tval =~ s/^\s+//g;
    $tval =~ s/\s+$//g;
    push @card_tag, $tval;
    }

    #rewind data file to beginning for second pass
    seek (DATA, 0, 0);

    # During second pass, "shift" the next (first) node from the array of
    "card_tags" and use it
    # as a unique identifier. Trigger "shift" based on line indicating
    each new card from CLI.
    # New card's currently start with "Card name"

    while ($line = <DATA>)
    {
    if ($line =~ /Card name/)
    {
    $port_count = -1; # zero based port count
    $card_key = shift @card_tag;
    }
    if ($line =~ /Port Id/)
    {
    $port_count++;
    }

    chomp($line);
    $line =~ /(.*?):(.*)/; # Delimiter ==> " : "
    $tkey = $1;
    $tval = $2;

    $tkey =~ s/^\s+//g; # Strip leading spaces from value
    $tkey =~ s/\s+$//g; # Strip trailing spaces from value

    $tval =~ s/^\s+//g;
    $tval =~ s/\s+$//g;

    $tval = "N/A" if (!$tval); # If "tval" is empty, assign value of
    "N/A"

    if ($port_count > -1) # Record port value
    {
    $card_hash->{$card_key}->{$port_count}->{$tkey}="$tval" if
    $tkey; #Load key & value; If blank line skip.
    }
    else # No port value
    {
    $card_hash->{$card_key}->{$tkey}="$tval" if $tkey; #Load
    key & value; If blank line skip.
    }
    }

    #Print hash to verfiy correct construction.
    print_hash($card_hash);

    ###########################################################
    # Print hash as deep as:
    # $pri_key=$sub_key=$sub_sub_key=$pri_hash->{$pri_key}->{$sub_key}->{$sub_sub_key}
    sub print_hash
    {
    my ($pri_hash) = @_ ;

    foreach my $pri_key ( keys %{$pri_hash} )
    {
    foreach my $sub_key ( keys %{ $pri_hash->{$pri_key} } )
    {
    if (ref($pri_hash->{$pri_key}->{$sub_key}) eq 'HASH' ) # ?
    "value" | embedded hash
    {
    foreach my $sub_sub_key ( keys %{
    $pri_hash->{$pri_key}->{$sub_key} } )
    {
    print "$pri_key = $sub_key = $sub_sub_key =
    $pri_hash->{$pri_key}->{$sub_key}->{$sub_sub_key}\n";
    }
    }
    else
    {
    print "$pri_key = $sub_key =
    $pri_hash->{$pri_key}->{$sub_key}\n";
    }
    }
    }
    }


    __END__

    <cli.log>

    Card name : SuperWombat
    Firmware Version : 4.234
    Driver Version : 7.2345
    Node Id : 00000
    Number of Ports : 2

    Port Id : 1a3c5f
    Port name : Alpha
    Port type : Ethernet
    Port speed : 10

    Port Id : af2c2d2
    Port name : Alpha2
    Port type : Scsi
    Port speed : 1gig

    Card name : BitBlaster
    Firmware Version : 5.000
    Driver Version : 8.000 rel 5.02
    Node Id : 24550
    Number of Ports : 1

    Port Id : 10200
    Port name : GigE1
    Port type : Ethernet
    Port speed : 1000
    disco, Aug 4, 2003
    #1
    1. Advertising

  2. disco wrote:
    >
    > I have output from a CLI that I am parsing into a data structure
    > (hash) to be used as input to another program which reads the hash.
    > The CLI always formats the data in the same by first listing the card
    > information followed by information for each port.
    >
    > I have figured out the routine to parse all of the data into a hash,
    > but what I can't figure out is how to do determine the unique key
    > value first by considering all of the CLI output and then use it as I
    > build the hash. There is probably a better way to avoid rewinding the
    > file as I've done too.
    >
    > Right now I just use the Node Id, but I really need my unique key to
    > be based on the following criteria:
    >
    > 1) If the "node id" <> 0's then use the Node ID as the unique
    > identifier.
    > 2) If the "node id" == 0's then use the port id with the smallest
    > alpha value (it is in hex) as the unique identifier.
    >
    > <cli.log>
    >
    > Card name : SuperWombat
    > Firmware Version : 4.234
    > Driver Version : 7.2345
    > Node Id : 00000
    > Number of Ports : 2
    >
    > Port Id : 1a3c5f
    > Port name : Alpha
    > Port type : Ethernet
    > Port speed : 10
    >
    > Port Id : af2c2d2
    > Port name : Alpha2
    > Port type : Scsi
    > Port speed : 1gig
    >
    > Card name : BitBlaster
    > Firmware Version : 5.000
    > Driver Version : 8.000 rel 5.02
    > Node Id : 24550
    > Number of Ports : 1
    >
    > Port Id : 10200
    > Port name : GigE1
    > Port type : Ethernet
    > Port speed : 1000
    >
    > Ideal output (understanding that in real life the hash doesn't come
    > out in order -- that doesn't matter, and that "->" is simply for
    > presentation and not part of data)
    >
    > 1a3c5f -> Card name -> SuperWombat
    > 1a3c5f -> Firmware Version -> 4.234
    > 1a3c5f -> Driver Version -> 7.2345
    > 1a3c5f -> Node Id -> 00000
    > 1a3c5f -> Number of Ports -> 2
    > 1a3c5f -> 0 -> Port Id -> 1a3c5f
    > 1a3c5f -> 0 -> Port name -> Alpha
    > 1a3c5f -> 0 -> Port type -> Ethernet
    > 1a3c5f -> 0 -> Port speed -> 10
    > 1a3c5f -> 1 -> Port Id -> af2c2d2
    > 1a3c5f -> 1 -> Port name -> Alpha2
    > 1a3c5f -> 1 -> Port type -> Scsi
    > 1a3c5f -> 1 -> Port speed -> 1gig
    > 24550 -> Card name -> BitBlaster
    > 24550 -> Firmware Version -> 5.000
    > 24550 -> Driver Version -> 8.000 rel 5.02
    > 24550 -> Node Id -> 24550
    > 24550 -> Number of Ports -> 1
    > 24550 -> 0 -> Port Id -> 10200
    > 24550 -> 0 -> Port name -> GigE1
    > 24550 -> 0 -> Port type -> Ethernet
    > 24550 -> 0 -> Port speed -> 1000
    >
    > Can anyone suggest a way to do this easily? I have spent alot of time
    > trying to push/pop/sort arrays, but I can't pull the whole thing
    > together. I know this can be done in Perl, but I am stuck.



    This produces your "Ideal output" format although it doesn't use a hash:

    #!/usr/bin/perl
    use warnings;
    use strict;

    my $file = 'cli.log';
    open my $fh, '<', $file or die "Cannot open $file: $!";

    local $/ = ''; # paragraph mode
    my @data;
    while ( <$fh> ) {
    my ( @temp, $nkey, $ports );
    for my $line ( grep [ s/^\s+//, s/\s+$// ], split /\n/ ) {
    my ( $key, $val ) = grep [ s/^\s+//, s/\s+$// ], split /:/, $line, 2;
    $nkey = $val if $key eq 'Node Id' and $val != 0;
    $ports = $val if $key eq 'Number of Ports';
    push @temp, [ $key, $val ];
    }

    my @nkeys;
    for my $port ( 1 .. $ports ) {
    for my $line ( grep [ s/^\s+//, s/\s+$// ], split /\n/, <$fh> ) {
    my ( $key, $val ) = grep [ s/^\s+//, s/\s+$// ], split /:/, $line, 2;
    push @nkeys, $val if $key eq 'Port Id';
    push @temp, [ $port - 1, $key, $val ];
    }
    }
    @nkeys = sort @nkeys;

    unshift @$_, defined $nkey ? $nkey : $nkeys[ 0 ] for @temp;
    push @data, @temp;
    }

    print join( ' -> ', @$_ ), "\n" for @data;

    __END__


    If you need the data in a hash this will do it.

    #!/usr/bin/perl
    use warnings;
    use strict;
    use Data::Dumper;

    my $file = 'cli.log';
    open my $fh, '<', $file or die "Cannot open $file: $!";

    local $/ = ''; # paragraph mode
    my %data;
    while ( <$fh> ) {
    my %temp = grep [ s/^\s+//, s/\s+$// ], split /:|\n/;

    my ( @keys, @ports );
    for my $port ( 1 .. $temp{ 'Number of Ports' } ) {
    push @ports, { grep [ s/^\s+//, s/\s+$// ], split /:|\n/, <$fh> };
    push @keys, $ports[ -1 ]{ 'Port Id' };
    }
    @keys = sort @keys;

    if ( $temp{ 'Node Id' } == 0 ) {
    $data{ $keys[ 0 ] } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
    }
    else {
    $data{ $temp{ 'Node Id' } } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
    }
    }

    print Dumper( \%data );

    __END__



    John
    --
    use Perl;
    program
    fulfillment
    John W. Krahn, Aug 4, 2003
    #2
    1. Advertising

  3. disco

    disco Guest

    "John W. Krahn" <> wrote in message >
    > If you need the data in a hash this will do it.
    >
    > #!/usr/bin/perl
    > use warnings;
    > use strict;
    > use Data::Dumper;
    >
    > my $file = 'cli.log';
    > open my $fh, '<', $file or die "Cannot open $file: $!";
    >
    > local $/ = ''; # paragraph mode
    > my %data;
    > while ( <$fh> ) {
    > my %temp = grep [ s/^\s+//, s/\s+$// ], split /:|\n/;
    >
    > my ( @keys, @ports );
    > for my $port ( 1 .. $temp{ 'Number of Ports' } ) {
    > push @ports, { grep [ s/^\s+//, s/\s+$// ], split /:|\n/, <$fh> };
    > push @keys, $ports[ -1 ]{ 'Port Id' };
    > }
    > @keys = sort @keys;
    >
    > if ( $temp{ 'Node Id' } == 0 ) {
    > $data{ $keys[ 0 ] } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
    > }
    > else {
    > $data{ $temp{ 'Node Id' } } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
    > }
    > }
    >
    > print Dumper( \%data );
    >
    > __END__
    >
    >
    >
    > John


    Hi John,

    Thanks for your ideas and feedback. I have gotten some good ideas and
    learned some new concepts already.

    I cannot run either of your examples with warnings turned on and the
    results are not correct. I cut and pasted your code onto my
    workstation. Is this the correct version of your code?

    I would like to get the second example working. This is what I get:

    Use of uninitialized value in sort at c:\temp\example2.pl line 19,
    <$fh> chunk 1.
    Use of uninitialized value in sort at c:\temp\example2.pl line 19,
    <$fh> chunk 1.
    Use of uninitialized value in sort at c:\temp\example2.pl line 19,
    <$fh> chunk 1.
    Use of uninitialized value in sort at c:\temp\example2.pl line 19,
    <$fh> chunk 1.
    Use of uninitialized value in hash element at c:\temp\example2.pl
    line 21, <$fh> chunk 1.
    Use of uninitialized value in hash element at c:\temp\example2.pl
    line 21, <$fh> chunk 1.
    $VAR1 = {
    '' => {
    '' => 'Card name',
    'Firmware Version' => '4.234',
    'BitBlaster' => 'Firmware Version',
    'Driver Version' => '7.2345',
    'Alpha' => 'Port type',
    'Ethernet' => 'Port speed',
    '5.000' => 'Driver Version',
    '24550' => 'Number of Ports',
    'Card name' => 'SuperWombat',
    'Node Id' => '00000',
    'Number of Ports' => 2,
    '0' => {},
    '1' => {},
    'Port name' => 'GigE1',
    '10' => '',
    '8.000 rel 5.02' => 'Node Id',
    'Port type' => 'Ethernet',
    'Port speed' => '1000',
    'Port Id' => '10200',
    '1a3c5f' => 'Port name'
    }
    };
    Press any key to continue . . .

    I was hoping for:

    1a3c5f -> Card name -> SuperWombat
    1a3c5f -> Firmware Version -> 4.234
    1a3c5f -> Driver Version -> 7.2345
    1a3c5f -> Node Id -> 00000
    1a3c5f -> Number of Ports -> 2
    1a3c5f -> 0 -> Port Id -> 1a3c5f
    1a3c5f -> 0 -> Port name -> Alpha
    1a3c5f -> 0 -> Port type -> Ethernet
    1a3c5f -> 0 -> Port speed -> 10
    1a3c5f -> 1 -> Port Id -> af2c2d2
    1a3c5f -> 1 -> Port name -> Alpha2
    1a3c5f -> 1 -> Port type -> Scsi
    1a3c5f -> 1 -> Port speed -> 1gig
    24550 -> Card name -> BitBlaster
    24550 -> Firmware Version -> 5.000
    24550 -> Driver Version -> 8.000 rel 5.02
    24550 -> Node Id -> 24550
    24550 -> Number of Ports -> 1
    24550 -> 0 -> Port Id -> 10200
    24550 -> 0 -> Port name -> GigE1
    24550 -> 0 -> Port type -> Ethernet
    24550 -> 0 -> Port speed -> 1000

    Note there are 2 separate cards and the special routine I can't figure
    out is for the card named "Super Wombat".

    Would there be any disadvatage to making "data" an anonymous hash? I
    like to use anonymous hashes because they are easy to add onto later.

    Thanks for your help,
    Disco
    disco, Aug 8, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric
    Replies:
    3
    Views:
    2,849
    OHM \( One Handed Man \)
    Apr 9, 2006
  2. Terry Jones
    Replies:
    3
    Views:
    781
    Nick Kew
    Mar 24, 2005
  3. =?ISO-8859-1?Q?Morris_Carr=E9?=

    Python passes the Turing test with a one-liner !

    =?ISO-8859-1?Q?Morris_Carr=E9?=, Apr 1, 2004, in forum: Python
    Replies:
    1
    Views:
    330
    Jon Perez
    Apr 2, 2004
  4. A. B., Khalid
    Replies:
    5
    Views:
    304
    A. B., Khalid
    Nov 22, 2004
  5. Alex van der Spek
    Replies:
    1
    Views:
    774
    Gabriel Genellina
    May 24, 2011
Loading...

Share This Page