Excessive Memory Usage with libxml

Discussion in 'Ruby' started by Daniel N, May 22, 2007.

  1. Daniel N

    Daniel N Guest

    Hi,

    I hope this is the right place for this.

    I'm writing a markup mangler that I'm using libxml for. Primarily for
    speed reasons.

    I've got some basic functionality going, it's not finished, and when
    I'm doing a benchmark I get some good figures.

    I'm using a recursive function to process my data. The general gist
    of it is as follows

    http://pastie.caboo.se/63584


    But...

    When I run the benchmark I get weird things happening.

    All benchmarks exhibit steadily increasing memory usage until the end
    of the benchmark
    10000 iterations were fine
    very much above this and I started to get

    malloc errors
    or
    segmentation fault


    I changed
    doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

    to
    doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

    taking a hit on performance, but it allowed me to get through 100,000
    iterations.
    The memory usage though was stupid at 235Mb Real and 233Mb Virtual

    The HTML I am parsing is a fairly small snippet

    I'm running on Mac OsX.

    Any ideas what and why?
    thankyou

    Daniel
    Daniel N, May 22, 2007
    #1
    1. Advertising

  2. Daniel N

    Daniel N Guest

    On 5/23/07, Daniel N <> wrote:
    > Hi,
    >
    > I hope this is the right place for this.
    >
    > I'm writing a markup mangler that I'm using libxml for. Primarily for
    > speed reasons.
    >
    > I've got some basic functionality going, it's not finished, and when
    > I'm doing a benchmark I get some good figures.
    >
    > I'm using a recursive function to process my data. The general gist
    > of it is as follows
    >
    > http://pastie.caboo.se/63584
    >
    >
    > But...
    >
    > When I run the benchmark I get weird things happening.
    >
    > All benchmarks exhibit steadily increasing memory usage until the end
    > of the benchmark
    > 10000 iterations were fine
    > very much above this and I started to get
    >
    > malloc errors
    > or
    > segmentation fault
    >
    >
    > I changed
    > doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )
    >
    > to
    > doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )
    >
    > taking a hit on performance, but it allowed me to get through 100,000
    > iterations.
    > The memory usage though was stupid at 235Mb Real and 233Mb Virtual
    >
    > The HTML I am parsing is a fairly small snippet
    >
    > I'm running on Mac OsX.
    >
    > Any ideas what and why?
    > thankyou
    >
    > Daniel
    >
    >


    I've managed to reproduce the error by changing to Benchmark.bmbm. It is

    ruby(3775) malloc: *** Deallocation of a pointer not malloced:
    0x3903a60; This could be a double free(), or free() called with the
    middle of an allocated block; Try setting environment variable
    MallocHelp to see tools to help debug
    ruby(3775) malloc: *** Deallocation of a pointer not malloced:
    0x75b0750; This could be a double free(), or free() called with the
    middle of an allocated block; Try setting environment variable
    MallocHelp to see tools to help debug
    ruby(3775) malloc: *** Deallocation of a pointer not malloced:
    0x75c4950; This could be a double free(), or free() called with the
    middle of an allocated block; Try setting environment variable
    MallocHelp to see tools to help debug
    /markup_mangler_benchmark.rb:79: [BUG] Segmentation fault
    Daniel N, May 22, 2007
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. metfan
    Replies:
    2
    Views:
    4,847
    Robert Olofsson
    Oct 21, 2003
  2. Ian Gregory
    Replies:
    1
    Views:
    503
  3. Olav
    Replies:
    3
    Views:
    4,238
  4. fft1976
    Replies:
    9
    Views:
    328
    fft1976
    Jul 1, 2009
  5. Chris
    Replies:
    9
    Views:
    149
    Chris
    Nov 20, 2006
Loading...

Share This Page