Memory Analysis after Hanging

Discussion in 'Perl Misc' started by Mark, Sep 4, 2003.

  1. Mark

    Mark Guest

    We are running a script to load text files into 8 MySQL tables. Each
    text file contains tens of thousands of lines, each line containing
    pipe seperated values. The script loops until all 8 tables are loaded
    and it has worked fine till now. For a new client we must upload
    nearly 10mb of data and our script gets threequarters of the way
    through the file and then just hangs. Our ISP confirms that the script
    has run out of memory.

    Does anyone know of any diagnostic tools that can help us to isolate
    the component(s) that are not being cleared from memory? As we use a
    shared hosting account through an ISP we don't have direct access to
    the server. I have been looking at the Devel::peek module but this
    seems to be for mod_perl only.

    The server has produced a core file, but we don't have telnet access
    to our shared server account. Are there any suitable readers that will
    run in windows?

    What is meant by including debug data in the appliciation in order to
    make the core file more readable?

    Is there an alternative way to solving this problem, e.g. if for each
    text file we opened a piped process using another perl script would
    memory be returned for further use once the piped process had
    terminated?


    Many thanks in advance,
    Mark
    Mark, Sep 4, 2003
    #1
    1. Advertising

  2. Mark

    John Bokma Guest

    Mark wrote:

    > We are running a script to load text files into 8 MySQL tables. Each
    > text file contains tens of thousands of lines, each line containing
    > pipe seperated values. The script loops until all 8 tables are loaded
    > and it has worked fine till now. For a new client we must upload
    > nearly 10mb of data and our script gets threequarters of the way
    > through the file and then just hangs. Our ISP confirms that the script
    > has run out of memory.


    Do you slurp the file or read it line by line? Do you store the info in
    each line or insert it on the fly?

    > Is there an alternative way to solving this problem, e.g. if for each


    post the script, unless it is huge. In that case provide us with a link.
    You can mail me the script if you want to keep it confidential but then
    I want a book in return :-D.

    --
    Kind regards, feel free to mail: mail(at)johnbokma.com (or reply)
    virtual home: http://johnbokma.com/ ICQ: 218175426
    John web site hints: http://johnbokma.com/websitedesign/
    John Bokma, Sep 4, 2003
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. qazmlp
    Replies:
    3
    Views:
    657
    Robert Olofsson
    Jan 7, 2004
  2. Alexander Hoffmann

    python memory analysis

    Alexander Hoffmann, Sep 6, 2004, in forum: Python
    Replies:
    1
    Views:
    312
    Alex Martelli
    Sep 6, 2004
  3. ssubbarayan
    Replies:
    5
    Views:
    2,287
    Dave Hansen
    Nov 3, 2009
  4. John Nagle

    feedparser hanging after I/O error

    John Nagle, Jun 1, 2011, in forum: Python
    Replies:
    2
    Views:
    330
    John Nagle
    Jun 3, 2011
  5. Thomas Leitner
    Replies:
    0
    Views:
    80
    Thomas Leitner
    May 23, 2008
Loading...

Share This Page