File slurp takes a long time on some files

[Tom Sliva]

I am using ActiveState Perl 5.6.1 on Win2k with 512Meg RAM. I have
two text files I am trying to slurp. File A is ~2.2M and 35k lines
and File B is ~1M and 43k lines (lines are shorter). I have two of
each file type that I read off the command line. I am slurping them
using the following code.

open (TESTFILE, $ARGV[0]) || die("could not open test file");
@testfile = <TESTFILE>;
close (TESFILE);

repeating for $ARGV[1] through 3 using different array names and
filehandles.

Type A files open and dump in ~1 second. Type B files take ~5 seconds
for the 1st file and ~77 seconds for the second. Additionally if I
rearrange the commandline so that all files read in are of type A, all
4 get read in ~4 seconds. If the commandline is changed to all type B
files, the 1st takes 5 seconds, the second ~77 seconds and the 3rd and
4th files take about 160-165 seconds each. These times stay fairly
consistent even if I open the same type B file 4 times in a row or run
the script immediately after a reboot.

There is no extra whitespace at the end of lines or the file of type B
files. I am out of ideas on what the cause could be.

any suggestions?
thanks,
--Tom

 

[Jim Keenan]

I am using ActiveState Perl 5.6.1 on Win2k with 512Meg RAM. I have
two text files I am trying to slurp. File A is ~2.2M and 35k lines
and File B is ~1M and 43k lines (lines are shorter). I have two of
each file type that I read off the command line. I am slurping them
using the following code.

open (TESTFILE, $ARGV[0]) || die("could not open test file");
@testfile = <TESTFILE>;
close (TESFILE);

I doubt that it's possible for us to help you knowing only the byte/line
counts of the two types of files. We'd have to see something of the
files themselves. And even then, the explanation for the problem may
have more to do with your OS than with Perl.

But here's a Perl question that probably occurred to many people reading
your question: Why are you trying to slurp these 4 large files into
memory? What problem are you trying to solve that requires that
approach as opposed to line-by-line processing with a 'while' loop?

Jim Keenan

 

[Tad McClellan]

I am slurping them
using the following code.

open (TESTFILE, $ARGV[0]) || die("could not open test file"); ^
@testfile = <TESTFILE>; ^
close (TESFILE);

^^
^^ huh?

 

[Joe Smith]

I have two text files I am trying to slurp.

Have you tried using File::Slurp instead of a do-it-yourself solution?
It is specifically optimized for certain cases.
-Joe

 

[Bart Lateur]

Type A files open and dump in ~1 second. Type B files take ~5 seconds
for the 1st file and ~77 seconds for the second.

Quick, uninformed guess: type B is too big to fit in the cache, type A
fits, though tightly.

 

[Uri Guttman]

JS> Have you tried using File::Slurp instead of a do-it-yourself solution?
JS> It is specifically optimized for certain cases.

ahem!! s/certain/all/



but the OP's problem sounds more like an OS/box issue than
software. such big differences in slurp time mean something else is
wrong. slurping a couple of megs or K's of lines should not take so
long.

uri

 

[Glenn Jackman]

I am using ActiveState Perl 5.6.1 on Win2k with 512Meg RAM. I have
two text files I am trying to slurp. File A is ~2.2M and 35k lines
and File B is ~1M and 43k lines (lines are shorter). I have two of [...]
Type A files open and dump in ~1 second. Type B files take ~5 seconds
for the 1st file and ~77 seconds for the second. Additionally if I

[...]

Are some of your files on a network drive, or are they all local?

 

[Tom Sliva]

JS> Have you tried using File::Slurp instead of a do-it-yourself solution?
JS> It is specifically optimized for certain cases.

ahem!! s/certain/all/

I was afraid it would be an OS issue, but File::Slurp did the trick.
I can now read all 4 files in about 2-3 seconds no matter what
combination. Thanks for everyone's help and thanks to Uri for writing
this package