why this code shots up memory usage

[Madhur]

Hello
The following code snippet, shots the memory usage to full and
results in Out of Memory error.
The substitution seems to be the culprit. But why it happens. I
am running W2K, Perl 5.6.

@input=<MYFILE>;
$i=0;
while($i<=@input)
{

$input[$i]=~s/^[0-9]+://;
print("$input[$i]");
$i++;
}
close(MYFILE);


--
Winners dont do different things, they do things differently.

Madhur Ahuja
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

 

[Jay Tilton]

: The following code snippet, shots the memory usage to full and
: results in Out of Memory error.
: The substitution seems to be the culprit. But why it happens. I
: am running W2K, Perl 5.6.
:
: @input=<MYFILE>;
: $i=0;
: while($i<=@input)
: {
:
: $input[$i]=~s/^[0-9]+://;
: print("$input[$i]");
: $i++;
: }
: close(MYFILE);

It happens because of autovivification. The s/// operation brings
$input[$i] into existence if it doesn't exist already. The loop will add
new elements to @input forever.

Change the condition either to:

while( $i < @input ) { ... }

or to:

while( $i <= $#input ) { ... }

or even better, completely ditch the index and directly iterate over the
array's elements:

for( @input ) {
s/^[0-9]+://;
print;
}

 

[John W. Krahn]

The following code snippet, shots the memory usage to full and
results in Out of Memory error.
The substitution seems to be the culprit. But why it happens. I
am running W2K, Perl 5.6.

@input=<MYFILE>;

You are reading the entire file into memory. If it is a large file then
this will die with an Out of Memory error.

$i=0;
while($i<=@input)
{

$input[$i]=~s/^[0-9]+://;
print("$input[$i]");
$i++;
}
close(MYFILE);

The usual way to write that in perl is:

while ( <MYFILE> ) {
s/^\d+://;
print;
}
close MYFILE;

Or another way: without the substitution:

while ( <MYFILE> ) {
print substr $_, 1 + index $_, ':';
}
close MYFILE;



John

 

[Madhur]

Hello
Thanks to you all.
But shouldn't the substitution $input[$i]=~s/^[0-9]+://;
terminate automatically after a newline. I remember there is a
option in
the substitution which lets you operate on multiple lines. Then
What's the use
of that switch?, if its skipping newlines already here.

It is because the line

$input[$i]=~s/^[0-9]+://;

keeps adding elements to the array after you have reached the end of the
data, so that

[snip]


When the end of data is reached, why the substitution is made, as
I have specified
the ^ operator, which specifies *the substitution should only
occur if the digit is found at the beginning*.


Or May be, I haven't understood the response.
--
Winners dont do different things, they do things differently.

Madhur Ahuja
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

The following code snippet, shots the memory usage to full and
results in Out of Memory error.
The substitution seems to be the culprit. But why it happens. I
am running W2K, Perl 5.6.

@input=<MYFILE>;

You are reading the entire file into memory. If it is a large file then
this will die with an Out of Memory error.

$i=0;
while($i<=@input)
{

$input[$i]=~s/^[0-9]+://;
print("$input[$i]");
$i++;
}
close(MYFILE);

The usual way to write that in perl is:

while ( <MYFILE> ) {
s/^\d+://;
print;
}
close MYFILE;

Or another way: without the substitution:

while ( <MYFILE> ) {
print substr $_, 1 + index $_, ':';
}
close MYFILE;



John

 

[Madhur]

This time the substitution is evaluated for $input[2], which does not
already exist. Perl creates $input[2] with an undef value. Now the array
has 3 elements.

[snip]

Oh! I didn't knew that. Thanks for clearing.
One more thing, Can I have remove the equal sign from the while
loop.

while($i<@input)

Would this be a good solution.


--
Winners dont do different things, they do things differently.

Madhur Ahuja
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

 

[Jay Tilton]

: One more thing, Can I have remove the equal sign from the while
: loop.
:
: while($i<@input)
:
: Would this be a good solution.

It will plug the hole that is sinking the program, but nobody competent
would call the program "good."

Recapping the original code, including that change:

@input=<MYFILE>;
$i=0;
while($i < @input) {
$input[$i] =~ s/^[0-9]+://;
print("$input[$i]");
$i++;
}
close(MYFILE);

The bad things about that:

1. Using global variables $i and @input instead of lexically declaring them
with my().

2. Useless use of quotes in the print() statement. See "What's wrong with
always quoting ``$vars''? " in perlfaq4.

3. Perl programmers would rather write "while($i < @input){...;$i++}" as
"for my $i (0..$#input){...}" .

4. But they would rather not do even that. Using the array elements'
indeces to iterate across the elements' values is distinctly anti-Perl.
The program treats all the array's values in the same way. Why should it
care where each element is located in the array?

5. The worst thing about the program is the array itself. Slurping an
entire file into memory just to process its records sequentially is amateur
programming. If you're processing sequentially, then read sequentially.

To see the program distilled into something good, reread the code given by
John W. Krahn in this thread.

while ( <MYFILE> ) {
s/^\d+://;
print;
}
close MYFILE;

 

[Madhur]

[snip]
The bad things about that:

1. Using global variables $i and @input instead of lexically declaring them
with my().

The code was part of main program rather than a procedure. Can i
use *my* in the main code?

2. Useless use of quotes in the print() statement. See "What's wrong with
always quoting ``$vars''? " in perlfaq4.

Indeed I agree, I won't never use them in future.

3. Perl programmers would rather write "while($i < @input){...;$i++}" as
"for my $i (0..$#input){...}" .

That means $i is local to the for loop. Right?

4. But they would rather not do even that. Using the array elements'
indeces to iterate across the elements' values is distinctly anti-Perl.
The program treats all the array's values in the same way. Why should it
care where each element is located in the array?

5. The worst thing about the program is the array itself. Slurping an
entire file into memory just to process its records sequentially is amateur
programming. If you're processing sequentially, then read sequentially.

To see the program distilled into something good, reread the code given by
John W. Krahn in this thread.

while ( <MYFILE> ) {
s/^\d+://;
print;
}
close MYFILE;

Could you please explain me,this. what does while(<MYFILE>)
means.

One more think, this is the code from the faq:
Stringification also destroys arrays.


@lines = `command`;
print "@lines"; # WRONG - extra blanks
print @lines; # right

Does it mean that value of @lines is changed. Will the @lines
contain now 2 elements 'command' and ' '.



--
Winners dont do different things, they do things differently.

Madhur Ahuja
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

 

[Jay Tilton]

:
: > while ( <MYFILE> ) {
: > s/^\d+://;
: > print;
: > }
: > close MYFILE;
:
: Could you please explain me,this. what does while(<MYFILE>)
: means.

This is explained in the "I/O Operators" subsection of perlop. The
"while(<MYFILE>) { ... }" construct reads single records from the MYFILE
filehandle into the $_ scalar until all records have been read.

: One more think, this is the code from the faq:
: Stringification also destroys arrays.
:
: @lines = `command`;
: print "@lines"; # WRONG - extra blanks
: print @lines; # right
:
: Does it mean that value of @lines is changed.

No. Stringification does not change the array in any way. It does change
the arguments given to a function, operator, or subroutine. For example:

foo( "@array" );

is equivalently written as:

foo( join($", @array) );

That passes a single string to the foo() subroutine. It is very different
from:

foo( @array )

which will pass a list containing the individual values from @array to the
foo() subroutine.

: Will the @lines contain now 2 elements 'command' and ' '.

No, but you misunderstand what the statement "@lines = `command`;" does.
The backticks are not normal quotes. They invoke an external command and
capture its output. See " qx/STRING/ " in the "Quote and Quote-like
Operators" subsection of perlop.

 

[Madhur]

Thank you very much!

--
Winners dont do different things, they do things differently.

Madhur Ahuja
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

 

[Madhur]

Thanks!
Another thing, I have seen this type of code:
for (@result){ print;}
for my $i(@result) {....}

As far as I know, for loop requires 3 arguments, and even if one
argument is given, the 2 semicolons must be there.
I have checked my book and the perl docs but haven't found the
reference about this.

--
Winners dont do different things, they do things differently.

Madhur
India
email : madhur<underscore>ahuja<at>yahoo<dot>com

 

[Ragnar Hafstað]

Thanks!
Another thing, I have seen this type of code:
for (@result){ print;}
for my $i(@result) {....}

As far as I know, for loop requires 3 arguments, and even if one
argument is given, the 2 semicolons must be there.
I have checked my book and the perl docs but haven't found the
reference about this.

these are called foreach loops
perldoc perlsyn

gnari

 

[Tad McClellan]

for (@result){ print;}

As far as I know, for loop requires 3 arguments, and even if one
argument is given, the 2 semicolons must be there.
I have checked my book and the perl docs but haven't found the
reference about this.

perldoc perlsyn

The C<foreach> keyword is actually a synonym for the C<for>
keyword, so you can use C<foreach> for readability or C<for>
for brevity.




[ Snip TOFU.
Please learn how to properly quote followups.
Soon.
]

 

[Michele Dondi]

Hello
The following code snippet, shots the memory usage to full and
results in Out of Memory error.
The substitution seems to be the culprit. But why it happens. I
am running W2K, Perl 5.6.

@input=<MYFILE>;
$i=0;
while($i<=@input)
{

$input[$i]=~s/^[0-9]+://;
print("$input[$i]");
$i++;
}
close(MYFILE);

s/^\d+://,print while <MYFILE>;


Michele

 

[Ben Morrow]

"You will never" or "you won't ever"...
;-)

There en't nuffin' wrong wi' dobble negitives..

Ben

 

[Michele Dondi]

The code was part of main program rather than a procedure. Can i
use *my* in the main code?

You *can* and, most importantly, you *should*. Wether the code is part
of a "procedure" or not is not relevant!

Indeed I agree, I won't never use them in future.

"You will never" or "you won't ever"...
;-)

Could you please explain me,this. what does while(<MYFILE>)
means.

You knew what @input=<MYFILE> means, now this is just about the same
thing apart that it is evaluated in scalar context and C<while> adds
some magic! But then, better than I could ever explain you, 'perldoc
perlop' does: search for "I/O Operators"!

One more think, this is the code from the faq:
Stringification also destroys arrays.

@lines = `command`;
print "@lines"; # WRONG - extra blanks
print @lines; # right

Does it mean that value of @lines is changed. Will the @lines
contain now 2 elements 'command' and ' '.

No and no.

Just try yourself (and do not bother with backticks!):

perl -le '@a=qw/a b c/; print "@a"; print @a;'


Michele

 

[Anno Siegel]

There en't nuffin' wrong wi' dobble negitives..

"Two negatives make a positive, but two positives never make a negative."

"Yeah, right!"

Anno

 

[Michele Dondi]

There en't nuffin' wrong wi' dobble negitives..

So I won't never correct them any more!


Michele