Need some hints on speeding up

[Guest]

I only do occasional Perl programming and most things I write are
short processes. I have something I'm working on that is scanning a
text file with about 15 million lines and trying to extract matches
from another text file, which has about 170 entries. The second text
file is read into an array. The process then scans through the big
file for certain possible patterns - it will find those in about 1 out
of 25 lines,, when it finds one, it then loops through the array
trying to find a match there, and then writes out a couple of lines
into another text file.

It also writes to the screen a summary line about every 25th record.

When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
almost everything.

But even 1.5 hours is too long since I need to be able to run this
with different sets of data several times a day,

One thought I had is that writting a summary progress to the screen
[which helps me judge how far along it is, may be slowing things down]

Running Windows 98 SE and the latest version of Active Perl.

 

[Jim Gibson]

I only do occasional Perl programming and most things I write are
short processes. I have something I'm working on that is scanning a
text file with about 15 million lines and trying to extract matches
from another text file, which has about 170 entries. The second text
file is read into an array. The process then scans through the big
file for certain possible patterns - it will find those in about 1 out
of 25 lines,, when it finds one, it then loops through the array
trying to find a match there, and then writes out a couple of lines
into another text file.

It also writes to the screen a summary line about every 25th record.

When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
almost everything.

But even 1.5 hours is too long since I need to be able to run this
with different sets of data several times a day,

One thought I had is that writting a summary progress to the screen
[which helps me judge how far along it is, may be slowing things down]

Running Windows 98 SE and the latest version of Active Perl.

People are going to need a little bit more information to help you.

First of all, comp.lang.perl is a defunct newsgroup. You would do
better to post to comp.lang.perl.misc.

What kind of hardware are you using? Your program may be limited by CPU
speed, disk I/O speed, or memory size. Can you upgrade your hardware if
that proves to be the limiting factor?

How long does it take to read through the 15M-line file? That gives you
a baseline for the minimum amount of time it will take to process the
file. If your full program takes considerably more than that, you may
be using a slow search algorithm or have other problems. Printing will
show down your program somewhat, but probably not a significant amount
unless you are really printing too much. Cut down the amount and see.

The best thing you can do is post a minimal, complete program (to
comp.lang.perl.misc) that people can inspect. Normally, you would want
a runnable program, but since you can't post the big text file or even
the 170 line pattern file, you may have to rely on code inspection
rather than profiling. Be sure and make your posted program readable.

You might want to profile your program. Check out 'perldoc -q profile'.