parsing log in multiple passes

[disco]

I have output from a CLI that I am parsing into a data structure
(hash) to be used as input to another program which reads the hash.
The CLI always formats the data in the same by first listing the card
information followed by information for each port.

I have figured out the routine to parse all of the data into a hash,
but what I can't figure out is how to do determine the unique key
value first by considering all of the CLI output and then use it as I
build the hash. There is probably a better way to avoid rewinding the
file as I've done too.

Right now I just use the Node Id, but I really need my unique key to
be based on the following criteria:

1) If the "node id" <> 0's then use the Node ID as the unique
identifier.
2) If the "node id" == 0's then use the port id with the smallest
alpha value (it is in hex) as the unique identifier.

<cli.log>

Card name : SuperWombat
Firmware Version : 4.234
Driver Version : 7.2345
Node Id : 00000
Number of Ports : 2

Port Id : 1a3c5f
Port name : Alpha
Port type : Ethernet
Port speed : 10

Port Id : af2c2d2
Port name : Alpha2
Port type : Scsi
Port speed : 1gig

Card name : BitBlaster
Firmware Version : 5.000
Driver Version : 8.000 rel 5.02
Node Id : 24550
Number of Ports : 1

Port Id : 10200
Port name : GigE1
Port type : Ethernet
Port speed : 1000

Ideal output (understanding that in real life the hash doesn't come
out in order -- that doesn't matter, and that "->" is simply for
presentation and not part of data)

1a3c5f -> Card name -> SuperWombat
1a3c5f -> Firmware Version -> 4.234
1a3c5f -> Driver Version -> 7.2345
1a3c5f -> Node Id -> 00000
1a3c5f -> Number of Ports -> 2
1a3c5f -> 0 -> Port Id -> 1a3c5f
1a3c5f -> 0 -> Port name -> Alpha
1a3c5f -> 0 -> Port type -> Ethernet
1a3c5f -> 0 -> Port speed -> 10
1a3c5f -> 1 -> Port Id -> af2c2d2
1a3c5f -> 1 -> Port name -> Alpha2
1a3c5f -> 1 -> Port type -> Scsi
1a3c5f -> 1 -> Port speed -> 1gig
24550 -> Card name -> BitBlaster
24550 -> Firmware Version -> 5.000
24550 -> Driver Version -> 8.000 rel 5.02
24550 -> Node Id -> 24550
24550 -> Number of Ports -> 1
24550 -> 0 -> Port Id -> 10200
24550 -> 0 -> Port name -> GigE1
24550 -> 0 -> Port type -> Ethernet
24550 -> 0 -> Port speed -> 1000


Can anyone suggest a way to do this easily? I have spent alot of time
trying to push/pop/sort arrays, but I can't pull the whole thing
together. I know this can be done in Perl, but I am stuck.

Thanks!
Disco

following is my code that shows what I have done so far...

#!/usr/bin/perl
use strict;

my $line; # Scalar to hold line from log file being processed.
my $card_hash = {}; #anonymous hash (data structure)
my $card_key; # Primary key
my $tkey; # Sub key
my $tval; # Value

my $port_count = 0;

my @temp;
my @card_tag;

open (DATA, "cli.log")or die("Cannot open: filename $!");

# Find node_wwn's to use as a unique identifier for each card.
# Create an array of node WWN's found and then rewind the file and
start again.
#

while (<DATA>)
{
chomp;
push @temp, $_ if /Node Id/; # get the entire line if it
contains "Node Id" and add it to array
}

foreach $line (@temp) # assign each item in list (array) to $line and
process
{
$line =~ /(.*?).*)/; # Delimiter ==> " : "
$tval = $2;
$tval =~ s/^\s+//g;
$tval =~ s/\s+$//g;
push @card_tag, $tval;
}

#rewind data file to beginning for second pass
seek (DATA, 0, 0);

# During second pass, "shift" the next (first) node from the array of
"card_tags" and use it
# as a unique identifier. Trigger "shift" based on line indicating
each new card from CLI.
# New card's currently start with "Card name"

while ($line = <DATA>)
{
if ($line =~ /Card name/)
{
$port_count = -1; # zero based port count
$card_key = shift @card_tag;
}
if ($line =~ /Port Id/)
{
$port_count++;
}

chomp($line);
$line =~ /(.*?).*)/; # Delimiter ==> " : "
$tkey = $1;
$tval = $2;

$tkey =~ s/^\s+//g; # Strip leading spaces from value
$tkey =~ s/\s+$//g; # Strip trailing spaces from value

$tval =~ s/^\s+//g;
$tval =~ s/\s+$//g;

$tval = "N/A" if (!$tval); # If "tval" is empty, assign value of
"N/A"

if ($port_count > -1) # Record port value
{
$card_hash->{$card_key}->{$port_count}->{$tkey}="$tval" if
$tkey; #Load key & value; If blank line skip.
}
else # No port value
{
$card_hash->{$card_key}->{$tkey}="$tval" if $tkey; #Load
key & value; If blank line skip.
}
}

#Print hash to verfiy correct construction.
print_hash($card_hash);

###########################################################
# Print hash as deep as:
# $pri_key=$sub_key=$sub_sub_key=$pri_hash->{$pri_key}->{$sub_key}->{$sub_sub_key}
sub print_hash
{
my ($pri_hash) = @_ ;

foreach my $pri_key ( keys %{$pri_hash} )
{
foreach my $sub_key ( keys %{ $pri_hash->{$pri_key} } )
{
if (ref($pri_hash->{$pri_key}->{$sub_key}) eq 'HASH' ) # ?
"value" | embedded hash
{
foreach my $sub_sub_key ( keys %{
$pri_hash->{$pri_key}->{$sub_key} } )
{
print "$pri_key = $sub_key = $sub_sub_key =
$pri_hash->{$pri_key}->{$sub_key}->{$sub_sub_key}\n";
}
}
else
{
print "$pri_key = $sub_key =
$pri_hash->{$pri_key}->{$sub_key}\n";
}
}
}
}


__END__

<cli.log>

Card name : SuperWombat
Firmware Version : 4.234
Driver Version : 7.2345
Node Id : 00000
Number of Ports : 2

Port Id : 1a3c5f
Port name : Alpha
Port type : Ethernet
Port speed : 10

Port Id : af2c2d2
Port name : Alpha2
Port type : Scsi
Port speed : 1gig

Card name : BitBlaster
Firmware Version : 5.000
Driver Version : 8.000 rel 5.02
Node Id : 24550
Number of Ports : 1

Port Id : 10200
Port name : GigE1
Port type : Ethernet
Port speed : 1000

 

[John W. Krahn]

I have output from a CLI that I am parsing into a data structure
(hash) to be used as input to another program which reads the hash.
The CLI always formats the data in the same by first listing the card
information followed by information for each port.

I have figured out the routine to parse all of the data into a hash,
but what I can't figure out is how to do determine the unique key
value first by considering all of the CLI output and then use it as I
build the hash. There is probably a better way to avoid rewinding the
file as I've done too.

Right now I just use the Node Id, but I really need my unique key to
be based on the following criteria:

1) If the "node id" <> 0's then use the Node ID as the unique
identifier.
2) If the "node id" == 0's then use the port id with the smallest
alpha value (it is in hex) as the unique identifier.

<cli.log>

Card name : SuperWombat
Firmware Version : 4.234
Driver Version : 7.2345
Node Id : 00000
Number of Ports : 2

Port Id : 1a3c5f
Port name : Alpha
Port type : Ethernet
Port speed : 10

Port Id : af2c2d2
Port name : Alpha2
Port type : Scsi
Port speed : 1gig

Card name : BitBlaster
Firmware Version : 5.000
Driver Version : 8.000 rel 5.02
Node Id : 24550
Number of Ports : 1

Port Id : 10200
Port name : GigE1
Port type : Ethernet
Port speed : 1000

Ideal output (understanding that in real life the hash doesn't come
out in order -- that doesn't matter, and that "->" is simply for
presentation and not part of data)

1a3c5f -> Card name -> SuperWombat
1a3c5f -> Firmware Version -> 4.234
1a3c5f -> Driver Version -> 7.2345
1a3c5f -> Node Id -> 00000
1a3c5f -> Number of Ports -> 2
1a3c5f -> 0 -> Port Id -> 1a3c5f
1a3c5f -> 0 -> Port name -> Alpha
1a3c5f -> 0 -> Port type -> Ethernet
1a3c5f -> 0 -> Port speed -> 10
1a3c5f -> 1 -> Port Id -> af2c2d2
1a3c5f -> 1 -> Port name -> Alpha2
1a3c5f -> 1 -> Port type -> Scsi
1a3c5f -> 1 -> Port speed -> 1gig
24550 -> Card name -> BitBlaster
24550 -> Firmware Version -> 5.000
24550 -> Driver Version -> 8.000 rel 5.02
24550 -> Node Id -> 24550
24550 -> Number of Ports -> 1
24550 -> 0 -> Port Id -> 10200
24550 -> 0 -> Port name -> GigE1
24550 -> 0 -> Port type -> Ethernet
24550 -> 0 -> Port speed -> 1000

Can anyone suggest a way to do this easily? I have spent alot of time
trying to push/pop/sort arrays, but I can't pull the whole thing
together. I know this can be done in Perl, but I am stuck.

This produces your "Ideal output" format although it doesn't use a hash:

#!/usr/bin/perl
use warnings;
use strict;

my $file = 'cli.log';
open my $fh, '<', $file or die "Cannot open $file: $!";

local $/ = ''; # paragraph mode
my @data;
while ( <$fh> ) {
my ( @temp, $nkey, $ports );
for my $line ( grep [ s/^\s+//, s/\s+$// ], split /\n/ ) {
my ( $key, $val ) = grep [ s/^\s+//, s/\s+$// ], split /:/, $line, 2;
$nkey = $val if $key eq 'Node Id' and $val != 0;
$ports = $val if $key eq 'Number of Ports';
push @temp, [ $key, $val ];
}

my @nkeys;
for my $port ( 1 .. $ports ) {
for my $line ( grep [ s/^\s+//, s/\s+$// ], split /\n/, <$fh> ) {
my ( $key, $val ) = grep [ s/^\s+//, s/\s+$// ], split /:/, $line, 2;
push @nkeys, $val if $key eq 'Port Id';
push @temp, [ $port - 1, $key, $val ];
}
}
@nkeys = sort @nkeys;

unshift @$_, defined $nkey ? $nkey : $nkeys[ 0 ] for @temp;
push @data, @temp;
}

print join( ' -> ', @$_ ), "\n" for @data;

__END__


If you need the data in a hash this will do it.

#!/usr/bin/perl
use warnings;
use strict;
use Data:umper;

my $file = 'cli.log';
open my $fh, '<', $file or die "Cannot open $file: $!";

local $/ = ''; # paragraph mode
my %data;
while ( <$fh> ) {
my %temp = grep [ s/^\s+//, s/\s+$// ], split /:|\n/;

my ( @keys, @ports );
for my $port ( 1 .. $temp{ 'Number of Ports' } ) {
push @ports, { grep [ s/^\s+//, s/\s+$// ], split /:|\n/, <$fh> };
push @keys, $ports[ -1 ]{ 'Port Id' };
}
@keys = sort @keys;

if ( $temp{ 'Node Id' } == 0 ) {
$data{ $keys[ 0 ] } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
}
else {
$data{ $temp{ 'Node Id' } } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
}
}

print Dumper( \%data );

__END__



John

 

[disco]

If you need the data in a hash this will do it.

#!/usr/bin/perl
use warnings;
use strict;
use Data:umper;

my $file = 'cli.log';
open my $fh, '<', $file or die "Cannot open $file: $!";

local $/ = ''; # paragraph mode
my %data;
while ( <$fh> ) {
my %temp = grep [ s/^\s+//, s/\s+$// ], split /:|\n/;

my ( @keys, @ports );
for my $port ( 1 .. $temp{ 'Number of Ports' } ) {
push @ports, { grep [ s/^\s+//, s/\s+$// ], split /:|\n/, <$fh> };
push @keys, $ports[ -1 ]{ 'Port Id' };
}
@keys = sort @keys;

if ( $temp{ 'Node Id' } == 0 ) {
$data{ $keys[ 0 ] } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
}
else {
$data{ $temp{ 'Node Id' } } = { %temp, map { $_, $ports[ $_ ] } 0 .. $#ports };
}
}

print Dumper( \%data );

__END__



John

Hi John,

Thanks for your ideas and feedback. I have gotten some good ideas and
learned some new concepts already.

I cannot run either of your examples with warnings turned on and the
results are not correct. I cut and pasted your code onto my
workstation. Is this the correct version of your code?

I would like to get the second example working. This is what I get:

Use of uninitialized value in sort at c:\temp\example2.pl line 19,
<$fh> chunk 1.
Use of uninitialized value in sort at c:\temp\example2.pl line 19,
<$fh> chunk 1.
Use of uninitialized value in sort at c:\temp\example2.pl line 19,
<$fh> chunk 1.
Use of uninitialized value in sort at c:\temp\example2.pl line 19,
<$fh> chunk 1.
Use of uninitialized value in hash element at c:\temp\example2.pl
line 21, <$fh> chunk 1.
Use of uninitialized value in hash element at c:\temp\example2.pl
line 21, <$fh> chunk 1.
$VAR1 = {
'' => {
'' => 'Card name',
'Firmware Version' => '4.234',
'BitBlaster' => 'Firmware Version',
'Driver Version' => '7.2345',
'Alpha' => 'Port type',
'Ethernet' => 'Port speed',
'5.000' => 'Driver Version',
'24550' => 'Number of Ports',
'Card name' => 'SuperWombat',
'Node Id' => '00000',
'Number of Ports' => 2,
'0' => {},
'1' => {},
'Port name' => 'GigE1',
'10' => '',
'8.000 rel 5.02' => 'Node Id',
'Port type' => 'Ethernet',
'Port speed' => '1000',
'Port Id' => '10200',
'1a3c5f' => 'Port name'
}
};
Press any key to continue . . .

I was hoping for:

1a3c5f -> Card name -> SuperWombat
1a3c5f -> Firmware Version -> 4.234
1a3c5f -> Driver Version -> 7.2345
1a3c5f -> Node Id -> 00000
1a3c5f -> Number of Ports -> 2
1a3c5f -> 0 -> Port Id -> 1a3c5f
1a3c5f -> 0 -> Port name -> Alpha
1a3c5f -> 0 -> Port type -> Ethernet
1a3c5f -> 0 -> Port speed -> 10
1a3c5f -> 1 -> Port Id -> af2c2d2
1a3c5f -> 1 -> Port name -> Alpha2
1a3c5f -> 1 -> Port type -> Scsi
1a3c5f -> 1 -> Port speed -> 1gig
24550 -> Card name -> BitBlaster
24550 -> Firmware Version -> 5.000
24550 -> Driver Version -> 8.000 rel 5.02
24550 -> Node Id -> 24550
24550 -> Number of Ports -> 1
24550 -> 0 -> Port Id -> 10200
24550 -> 0 -> Port name -> GigE1
24550 -> 0 -> Port type -> Ethernet
24550 -> 0 -> Port speed -> 1000

Note there are 2 separate cards and the special routine I can't figure
out is for the card named "Super Wombat".

Would there be any disadvatage to making "data" an anonymous hash? I
like to use anonymous hashes because they are easy to add onto later.

Thanks for your help,
Disco