Out of memory! When running perl script on windows

Discussion in 'Perl Misc' started by felad, Aug 27, 2006.

  1. felad

    felad Guest

    Hi

    I have Perl program on windows that suppose to run on about 10,000
    items in a loop ( should take 2-3 days for it to run )
    The problem is that after several hours I get this error:
    Out of memory!
    Callback called exit.

    I have found that every single loop the commit memory grow by 6000 K so
    I guess this is why the program crash
    But I really can't find that cause of it, most of my variable are local
    and I can't find any infinite loop

    Any ideas ?
    felad, Aug 27, 2006
    #1
    1. Advertising

  2. felad

    Matt Garrish Guest

    felad wrote:

    >
    > I have Perl program on windows that suppose to run on about 10,000
    > items in a loop ( should take 2-3 days for it to run )
    > The problem is that after several hours I get this error:
    > Out of memory!
    > Callback called exit.
    >
    > I have found that every single loop the commit memory grow by 6000 K so
    > I guess this is why the program crash
    > But I really can't find that cause of it, most of my variable are local
    > and I can't find any infinite loop
    >
    > Any ideas ?


    Why does it take 2-3 days to run 10,000 items? What is it that you are
    doing? Are you calling some other program that has a memory leak? Are
    you appending data to a variable that doesn't get reset with each
    iteration of the loop?

    It's nearly impossible to give you any meaningful advice if you don't
    give specifics and show some code.

    Matt
    Matt Garrish, Aug 27, 2006
    #2
    1. Advertising

  3. On 2006-08-27 15:30, felad <> wrote:
    > I have found that every single loop the commit memory grow by 6000 K so
    > I guess this is why the program crash
    > But I really can't find that cause of it, most of my variable are local
    > and I can't find any infinite loop
    >
    > Any ideas ?


    First, check that you aren't creating cyclic data structures: The perl
    garbage collector cannot free cyclic data structures. If you really need
    them, use weak references or destroy them explicitely after you are done
    with them.

    Second, lexical variables aren't freed immediately after they go out of
    scope but only after the function returns. So if you create large
    temporary variables, use them inside functions, not in the main program.

    Third, modules which may be helpful: Devel::Cycle, Devel::Leak.

    Fourth, perl has quite a bit overhead for each object: If you must keep
    lots of data in memory, try to keep few large objects instead of many
    little ones.

    hp

    --
    _ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
    |_|_) | Sysadmin WSR | > ist?
    | | | | Was sonst wäre der Sinn des Erfindens?
    __/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
    Peter J. Holzer, Aug 27, 2006
    #3
  4. Hi
    i used to do scripting for Bioinformatics and still doing it. Most of
    the task is to swallow a huge chunk of text, parse it, then tranfer or
    convert into another form (SQL database, other text format etc...). I
    had met this kind of problem several times.

    Normally what i do is :
    1- check for infinite loop, which caused by
    + unchanged variable values (missing $i++ etc.)
    + never fulfilled condition (-1 > 0 etc.)
    2- undef not-used variable in the middle of script (sometimes these
    variables hold huge data from file, and used only before parsing). So
    undef after parsing, and before doing other task, to save memory.
    3- optimize (reading file chunk-by-chunk can save memory than slurping
    a whole file etc.)

    etc.




    Peter J. Holzer wrote:
    > On 2006-08-27 15:30, felad <> wrote:
    > > I have found that every single loop the commit memory grow by 6000 K so
    > > I guess this is why the program crash
    > > But I really can't find that cause of it, most of my variable are local
    > > and I can't find any infinite loop
    > >
    > > Any ideas ?

    >
    > First, check that you aren't creating cyclic data structures: The perl
    > garbage collector cannot free cyclic data structures. If you really need
    > them, use weak references or destroy them explicitely after you are done
    > with them.
    >
    > Second, lexical variables aren't freed immediately after they go out of
    > scope but only after the function returns. So if you create large
    > temporary variables, use them inside functions, not in the main program.
    >
    > Third, modules which may be helpful: Devel::Cycle, Devel::Leak.
    >
    > Fourth, perl has quite a bit overhead for each object: If you must keep
    > lots of data in memory, try to keep few large objects instead of many
    > little ones.
    >
    > hp
    >
    > --
    > _ | Peter J. Holzer | > Wieso sollte man etwas erfinden was nicht
    > |_|_) | Sysadmin WSR | > ist?
    > | | | | Was sonst wäre der Sinn des Erfindens?
    > __/ | http://www.hjp.at/ | -- P. Einstein u. V. Gringmuth in desd
    alpha_beta_release, Aug 28, 2006
    #4
  5. felad

    Klaus Guest

    felad wrote:
    > I have Perl program on windows that suppose to run on about 10,000
    > items in a loop ( should take 2-3 days for it to run )
    > The problem is that after several hours I get this error:
    > Out of memory!
    > Callback called exit.
    >
    > I have found that every single loop the commit memory grow by 6000 K so
    > I guess this is why the program crash
    > But I really can't find that cause of it, most of my variable are local
    > and I can't find any infinite loop


    Can you post some code ?

    (that might not be easy: preferrably a short but complete version of
    the program, that still causes the Out of memory error)
    Klaus, Aug 28, 2006
    #5
  6. felad

    felad Guest

    Hi

    The script is very long so I don't think it will help
    Still couldn't find the problem but now I was thinking about printing
    how much memory the program \ process takes in several functions to try
    to find the problem
    Which perl module I can use to print the memory the script takes ?

    Thanks
    felad, Aug 28, 2006
    #6
  7. felad

    Dr.Ruud Guest

    alpha_beta_release schreef:

    > 2- undef not-used variable in the middle of script (sometimes these
    > variables hold huge data from file, and used only before parsing).
    > So undef after parsing, and before doing other task, to save memory.


    The variable needs to get out of scope AND not being referenced anymore,
    for the memory to be returned.

    perldoc -q memory
    perldoc -q shrink

    --
    Affijn, Ruud

    "Gewoon is een tijger."
    Dr.Ruud, Aug 28, 2006
    #7
  8. felad

    Guest

    "felad" <> wrote:
    > Hi
    >
    > I have Perl program on windows that suppose to run on about 10,000
    > items in a loop ( should take 2-3 days for it to run )


    Wow, that is a long time for 10,000 items. What are you doing to them?

    > The problem is that after several hours I get this error:
    > Out of memory!
    > Callback called exit.
    >
    > I have found that every single loop the commit memory grow by 6000 K so
    > I guess this is why the program crash
    > But I really can't find that cause of it, most of my variable are local
    > and I can't find any infinite loop
    >
    > Any ideas ?


    It is hard to say without knowing more. If each item is processed
    seperately (i.e. results of current item don't depend on results of past
    items) then it should be easy to add a "next" or "return" part way through
    the processing code, so that each item is only partially processed. Move
    the next or return statement around in a binary search-like way to figure
    out where the leak is triggered. Montior the memory useage closely so that
    you can figure out if it is leaking without having to wait for 3 days
    and/or for a OOM error.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
    , Aug 28, 2006
    #8
  9. Ah yes, that for automatic reallocation (using closure {}). To tell
    Perl explicitly ('we are done with this thing here') we use undef. BTW
    i'm also suggesting to try multi threading.

    Dr.Ruud wrote:
    > alpha_beta_release schreef:
    >
    > > 2- undef not-used variable in the middle of script (sometimes these
    > > variables hold huge data from file, and used only before parsing).
    > > So undef after parsing, and before doing other task, to save memory.

    >
    > The variable needs to get out of scope AND not being referenced anymore,
    > for the memory to be returned.
    >
    > perldoc -q memory
    > perldoc -q shrink
    >
    > --
    > Affijn, Ruud
    >
    > "Gewoon is een tijger."
    alpha_beta_release, Aug 30, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. dpackwood
    Replies:
    3
    Views:
    1,784
  2. Marcus Brody

    My perl script is "Killed" - Ran out of memory

    Marcus Brody, Sep 5, 2003, in forum: Perl Misc
    Replies:
    6
    Views:
    125
    Marcus Brody
    Sep 11, 2003
  3. Pea
    Replies:
    8
    Views:
    143
    Tad McClellan
    Jan 21, 2004
  4. Replies:
    4
    Views:
    183
    Josef Moellers
    Oct 27, 2007
  5. Replies:
    4
    Views:
    292
    ccc31807
    Jul 29, 2011
Loading...

Share This Page