My PERL program slows down over time.

Discussion in 'Perl Misc' started by Marc, Mar 4, 2004.

  1. Marc

    Marc Guest

    I wrote a PERL program to creat thousands of files. The program zips
    along at first but then begins to slow down.

    Does any one have any ideas on what might be causing this?

    -M
     
    Marc, Mar 4, 2004
    #1
    1. Advertising

  2. Marc

    gnari Guest

    "Marc" <> wrote in message
    news:...
    > I wrote a PERL program to creat thousands of files. The program zips
    > along at first but then begins to slow down.
    >
    > Does any one have any ideas on what might be causing this?


    yes. you are doing at leas 3 things wrong:

    a) you are createing all these files in the same directory, and your filing
    system
    slows down as the number increases

    b) you use a fancy method to make sure the file does not exist already.
    for my $fileno (1....10000) {
    $subnum=0;
    $subnum++ while (-f "file_$subnum");
    # create file
    }
    you have to do more and more work for each file
    if you had called your file "file_$fileno"."_$subnum" instead,
    you would not have had this problem

    c) the logic error on line 42 makes you create file number n
    n times.

    gnari
     
    gnari, Mar 4, 2004
    #2
    1. Advertising

  3. Marc

    Sam Holden Guest

    On 3 Mar 2004 23:37:28 -0800, Marc <> wrote:
    > I wrote a PERL program to creat thousands of files. The program zips
    > along at first but then begins to slow down.
    >
    > Does any one have any ideas on what might be causing this?


    Most file systems use a linear search through directory entries. Hence file
    system operations (opening a file, deleting a file, creating a file, and so
    on) slow down as the number of directory entries increase.

    Use a file system with a better than linear data structure/algorithm or
    create subdirectories based on sum property of the filename (a hash, or
    the first letter are commonly used).

    --
    Sam Holden
     
    Sam Holden, Mar 4, 2004
    #3
  4. Marc wrote:
    > I wrote a PERL program to creat thousands of files. The program zips
    > along at first but then begins to slow down.
    >
    > Does any one have any ideas on what might be causing this?


    Your while loop on line 42 causes a memory leak.

    jue
     
    Jürgen Exner, Mar 4, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    1,858
    Scott Allen
    Dec 9, 2005
  2. Elhanan

    java slows down suddenly?

    Elhanan, Apr 28, 2006, in forum: Java
    Replies:
    1
    Views:
    2,243
    Jade Yuan
    Apr 29, 2006
  3. mamin

    Application slows down

    mamin, May 9, 2006, in forum: ASP .Net
    Replies:
    3
    Views:
    387
    mamin
    May 11, 2006
  4. Jose Walker
    Replies:
    1
    Views:
    327
    =?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=
    Jul 13, 2006
  5. Replies:
    10
    Views:
    314
    BartC
    Sep 3, 2013
Loading...

Share This Page