Need some hints on speeding up

Discussion in 'Perl' started by Spamtrap, Aug 11, 2004.

  1. Spamtrap

    Spamtrap Guest

    I only do occasional Perl programming and most things I write are
    short processes. I have something I'm working on that is scanning a
    text file with about 15 million lines and trying to extract matches
    from another text file, which has about 170 entries. The second text
    file is read into an array. The process then scans through the big
    file for certain possible patterns - it will find those in about 1 out
    of 25 lines,, when it finds one, it then loops through the array
    trying to find a match there, and then writes out a couple of lines
    into another text file.

    It also writes to the screen a summary line about every 25th record.

    When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
    to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
    almost everything.

    But even 1.5 hours is too long since I need to be able to run this
    with different sets of data several times a day,

    One thought I had is that writting a summary progress to the screen
    [which helps me judge how far along it is, may be slowing things down]

    Running Windows 98 SE and the latest version of Active Perl.
    Spamtrap, Aug 11, 2004
    #1
    1. Advertising

  2. Spamtrap

    Jim Gibson Guest

    In article <>, Spamtrap
    <> wrote:

    > I only do occasional Perl programming and most things I write are
    > short processes. I have something I'm working on that is scanning a
    > text file with about 15 million lines and trying to extract matches
    > from another text file, which has about 170 entries. The second text
    > file is read into an array. The process then scans through the big
    > file for certain possible patterns - it will find those in about 1 out
    > of 25 lines,, when it finds one, it then loops through the array
    > trying to find a match there, and then writes out a couple of lines
    > into another text file.
    >
    > It also writes to the screen a summary line about every 25th record.
    >
    > When I run this it takes anywhere from 1.5 to 7.5 hours. It seems that
    > to avoid the 7.5 hour time, I should fresh reboot, and ctrl+alt+del
    > almost everything.
    >
    > But even 1.5 hours is too long since I need to be able to run this
    > with different sets of data several times a day,
    >
    > One thought I had is that writting a summary progress to the screen
    > [which helps me judge how far along it is, may be slowing things down]
    >
    > Running Windows 98 SE and the latest version of Active Perl.


    People are going to need a little bit more information to help you.

    First of all, comp.lang.perl is a defunct newsgroup. You would do
    better to post to comp.lang.perl.misc.

    What kind of hardware are you using? Your program may be limited by CPU
    speed, disk I/O speed, or memory size. Can you upgrade your hardware if
    that proves to be the limiting factor?

    How long does it take to read through the 15M-line file? That gives you
    a baseline for the minimum amount of time it will take to process the
    file. If your full program takes considerably more than that, you may
    be using a slow search algorithm or have other problems. Printing will
    show down your program somewhat, but probably not a significant amount
    unless you are really printing too much. Cut down the amount and see.

    The best thing you can do is post a minimal, complete program (to
    comp.lang.perl.misc) that people can inspect. Normally, you would want
    a runnable program, but since you can't post the big text file or even
    the 170 line pattern file, you may have to rely on code inspection
    rather than profiling. Be sure and make your posted program readable.

    You might want to profile your program. Check out 'perldoc -q profile'.
    Jim Gibson, Aug 12, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gertraud
    Replies:
    2
    Views:
    396
    Gertraud
    Mar 17, 2005
  2. Thin Myrna
    Replies:
    0
    Views:
    433
    Thin Myrna
    Jan 3, 2008
  3. erikcw
    Replies:
    2
    Views:
    249
    Arnaud Delobelle
    Oct 30, 2008
  4. Replies:
    2
    Views:
    443
  5. Corobori

    Need some hints before starting to change a page

    Corobori, Mar 2, 2005, in forum: ASP .Net Datagrid Control
    Replies:
    1
    Views:
    103
    Ken Cox [Microsoft MVP]
    Mar 4, 2005
Loading...

Share This Page