What to do about memory leaks

Discussion in 'Perl Misc' started by Martin Trautmann, Jul 20, 2007.

  1. Hi all,

    I got a perl script here which takes more memory than expected: It
    starts at about 10 MB, which is ok. After half of the job (3000 URL
    checks) it got > 500 MB, further increasing up to its termination. I've
    seen several segmentation fault errors as soon as the script would end,
    but don't know why. It's just a while loop, which is processed again and
    again without problems, while there's no more command after the end of
    this loop. Maybe the increased memory size is related to this error.

    I use strict and warnings, I verified that there was nothing wrong about
    the variables and open/close files.

    I take extensive use of WWW::Mechanize, without any knowledge what going
    on inside.

    I used Devel::Leaks in order to get the info that there's much going on -
    but I do not know yet what to do about this kind of information.

    What's your recommendation for a perl novice how to find out about these
    leaks?

    Thanks,
    Martin
    Martin Trautmann, Jul 20, 2007
    #1
    1. Advertising

  2. Martin Trautmann

    -berlin.de Guest

    Martin Trautmann <> wrote in comp.lang.perl.misc:
    >
    > Hi all,
    >
    > I got a perl script here which takes more memory than expected: It
    > starts at about 10 MB, which is ok. After half of the job (3000 URL
    > checks) it got > 500 MB, further increasing up to its termination. I've
    > seen several segmentation fault errors as soon as the script would end,
    > but don't know why. It's just a while loop, which is processed again and
    > again without problems, while there's no more command after the end of
    > this loop. Maybe the increased memory size is related to this error.
    >
    > I use strict and warnings, I verified that there was nothing wrong about
    > the variables and open/close files.
    >
    > I take extensive use of WWW::Mechanize, without any knowledge what going
    > on inside.
    >
    > I used Devel::Leaks in order to get the info that there's much going on -
    > but I do not know yet what to do about this kind of information.


    What's that? I can't find it on CPAN.

    > What's your recommendation for a perl novice how to find out about these
    > leaks?


    None.

    We'd have to see more of your code. Reduce it to a minimal example
    that still exhibits the behavior and post that.

    Anno
    -berlin.de, Jul 20, 2007
    #2
    1. Advertising

  3. On 20 Jul 2007 11:35:00 GMT, -berlin.de wrote:
    > Martin Trautmann <> wrote in comp.lang.perl.misc:
    > >
    > > Hi all,
    > >
    > > I got a perl script here which takes more memory than expected: It
    > > starts at about 10 MB, which is ok. After half of the job (3000 URL
    > > checks) it got > 500 MB, further increasing up to its termination. I've
    > > seen several segmentation fault errors as soon as the script would end,
    > > but don't know why. It's just a while loop, which is processed again and
    > > again without problems, while there's no more command after the end of
    > > this loop. Maybe the increased memory size is related to this error.
    > >
    > > I use strict and warnings, I verified that there was nothing wrong about
    > > the variables and open/close files.
    > >
    > > I take extensive use of WWW::Mechanize, without any knowledge what going
    > > on inside.
    > >
    > > I used Devel::Leaks in order to get the info that there's much going on -
    > > but I do not know yet what to do about this kind of information.

    >
    > What's that? I can't find it on CPAN.


    The module itself is Devel::Leak
    http://search.cpan.org/~ni-s/Devel-Leak-0.03/

    > > What's your recommendation for a perl novice how to find out about these
    > > leaks?

    >
    > None.
    >
    > We'd have to see more of your code. Reduce it to a minimal example
    > that still exhibits the behavior and post that.


    I don have a short example at hand - it's an interactive web site where
    I have to step along some inputs up to the result, then start again.

    I'll try to reproduce it while submitting it to a local server and find
    out more about it...

    The script is not that long, if I should mail it to you completely.

    Thanks,
    Martin
    Martin Trautmann, Jul 20, 2007
    #3
  4. On 20 Jul 2007 11:45:40 GMT, Abigail wrote:
    > Check line 17.


    [http://search.cpan.org/~ni-s/Devel-Leak-0.03/|Devel::Leak]s

    my perl was NOT build with -DDEBUGGING and compilation warns me about an
    existing sv_dump. I don't know what I may expect from this Utitlity - I
    get the memory warnings, but I don't know yet what to do with them.

    What should I learn from the result of Devel::Leak::CheckSV($handle);

    new 0x18ebd08 :

    Thanks,
    Martin
    Martin Trautmann, Jul 20, 2007
    #4
  5. Martin Trautmann

    -berlin.de Guest

    Martin Trautmann <> wrote in comp.lang.perl.misc:
    > On 20 Jul 2007 11:35:00 GMT, -berlin.de wrote:
    > > Martin Trautmann <> wrote in comp.lang.perl.misc:
    > > >
    > > > Hi all,
    > > >
    > > > I got a perl script here which takes more memory than expected: It
    > > > starts at about 10 MB, which is ok. After half of the job (3000 URL
    > > > checks) it got > 500 MB, further increasing up to its termination. I've
    > > > seen several segmentation fault errors as soon as the script would end,
    > > > but don't know why. It's just a while loop, which is processed again and
    > > > again without problems, while there's no more command after the end of
    > > > this loop. Maybe the increased memory size is related to this error.
    > > >
    > > > I use strict and warnings, I verified that there was nothing wrong about
    > > > the variables and open/close files.
    > > >
    > > > I take extensive use of WWW::Mechanize, without any knowledge what going
    > > > on inside.
    > > >
    > > > I used Devel::Leaks in order to get the info that there's much going on -
    > > > but I do not know yet what to do about this kind of information.

    > >
    > > What's that? I can't find it on CPAN.

    >
    > The module itself is Devel::Leak
    > http://search.cpan.org/~ni-s/Devel-Leak-0.03/


    Aha. Please be careful, typos like that frustrate those who try to help
    you.

    > > > What's your recommendation for a perl novice how to find out about these
    > > > leaks?

    > >
    > > None.
    > >
    > > We'd have to see more of your code. Reduce it to a minimal example
    > > that still exhibits the behavior and post that.

    >
    > I don have a short example at hand - it's an interactive web site where
    > I have to step along some inputs up to the result, then start again.
    >
    > I'll try to reproduce it while submitting it to a local server and find
    > out more about it...
    >
    > The script is not that long, if I should mail it to you completely.


    No, not to me personally.

    Read the posting guidelines that come up here twice a week. It has
    recommendations on how to prepare an example script. When you have
    a short demo of the problem, post it here. You'll probably find
    the solution while preparing the demo, it often happens.

    Anno
    -berlin.de, Jul 20, 2007
    #5
  6. On 20 Jul 2007 12:19:08 GMT, -berlin.de wrote:
    > > The module itself is Devel::Leak
    > > http://search.cpan.org/~ni-s/Devel-Leak-0.03/

    >
    > Aha. Please be careful, typos like that frustrate those who try to help
    > you.


    Sorry about that - I may be used too much to wiki styles, changing suffix or
    conjugation for better readability

    > Read the posting guidelines that come up here twice a week. It has
    > recommendations on how to prepare an example script. When you have
    > a short demo of the problem, post it here. You'll probably find
    > the solution while preparing the demo, it often happens.


    Ok, I´ll try a combination of a cgi perl script and a mechanize script,
    asking the same page again and again and will make these scripts more
    and more complex until they will show the leaking - if they do.


    I hoped that there are other modules that would give more information
    what's going on. Devel::Leak does show me that something is leaking -
    now I´d like to see further info what to do about it. This package from
    2000 is somehow dead and unmaintained (unfortunately the owner also died
    himself), while there's little docu for the novice what to do with this
    kind of info.
    Martin Trautmann, Jul 20, 2007
    #6
  7. Martin Trautmann

    baalbek Guest

    Martin Trautmann wrote:

    > I take extensive use of WWW::Mechanize, without any knowledge what going
    > on inside.


    Do remember that WWW::Mechanize caches the fetched web pages by default,
    thus using a lot of memory if you access a lot of web pages.

    In order not to cache the downloaded result, use this constructor:

    my $agent = WWW::Mechanize->new(stack_depth => 1);

    that is, stack_depth is set to one, so no caching.

    Hope this helps.

    ~Christian
    baalbek, Jul 20, 2007
    #7
  8. On Sat, 21 Jul 2007 00:33:53 +0200, baalbek wrote:
    > In order not to cache the downloaded result, use this constructor:
    >
    > my $agent = WWW::Mechanize->new(stack_depth => 1);


    Thanks!
    Martin
    Martin Trautmann, Jul 21, 2007
    #8
  9. Abigail wrote:
    > _
    > Martin Trautmann () wrote on VLXXI September MCMXCIII in
    > <URL:news:>:
    > ()
    > () Hi all,
    > ()
    > () I got a perl script here which takes more memory than expected: It
    > () starts at about 10 MB, which is ok. After half of the job (3000
    > URL () checks) it got > 500 MB, further increasing up to its
    > termination. I've () seen several segmentation fault errors as soon
    > as the script would end, () but don't know why. It's just a while
    > loop, which is processed again and () again without problems, while
    > there's no more command after the end of () this loop. Maybe the
    > increased memory size is related to this error. ()
    > () I use strict and warnings, I verified that there was nothing
    > wrong about () the variables and open/close files.
    > ()
    > () I take extensive use of WWW::Mechanize, without any knowledge
    > what going () on inside.
    > ()
    > () I used Devel::Leaks in order to get the info that there's much
    > going on - () but I do not know yet what to do about this kind of
    > information. ()
    > () What's your recommendation for a perl novice how to find out
    > about these () leaks?
    >
    >
    > Check line 17.


    Check quoting delimiters.

    --
    CL
    Clenna Lumina, Jul 23, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ASP.Confused

    ASP.NET - Detecting memory leaks

    ASP.Confused, Jul 16, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    2,738
    Marina
    Jul 16, 2004
  2. =?Utf-8?B?RnJhbmsxMjEz?=

    Memory Leaks in ASP.NET

    =?Utf-8?B?RnJhbmsxMjEz?=, Mar 4, 2005, in forum: ASP .Net
    Replies:
    12
    Views:
    13,841
  3. James Hunter Ross

    Our memory leaks?

    James Hunter Ross, Oct 20, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    771
    Bruce Barker
    Oct 20, 2005
  4. Novice
    Replies:
    28
    Views:
    5,095
    Jon Skeet
    Jul 22, 2003
  5. Replies:
    4
    Views:
    118
Loading...

Share This Page