Excessive Memory Usage with libxml

[Daniel N]

Hi,

I hope this is the right place for this.

I'm writing a markup mangler that I'm using libxml for. Primarily for
speed reasons.

I've got some basic functionality going, it's not finished, and when
I'm doing a benchmark I get some good figures.

I'm using a recursive function to process my data. The general gist
of it is as follows

http://pastie.caboo.se/63584


But...

When I run the benchmark I get weird things happening.

All benchmarks exhibit steadily increasing memory usage until the end
of the benchmark
10000 iterations were fine
very much above this and I started to get

malloc errors
or
segmentation fault


I changed
doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

to
doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

taking a hit on performance, but it allowed me to get through 100,000
iterations.
The memory usage though was stupid at 235Mb Real and 233Mb Virtual

The HTML I am parsing is a fairly small snippet

I'm running on Mac OsX.

Any ideas what and why?
thankyou

Daniel

 

[Daniel N]

Hi,

I hope this is the right place for this.

I'm writing a markup mangler that I'm using libxml for. Primarily for
speed reasons.

I've got some basic functionality going, it's not finished, and when
I'm doing a benchmark I get some good figures.

I'm using a recursive function to process my data. The general gist
of it is as follows

http://pastie.caboo.se/63584


But...

When I run the benchmark I get weird things happening.

All benchmarks exhibit steadily increasing memory usage until the end
of the benchmark
10000 iterations were fine
very much above this and I started to get

malloc errors
or
segmentation fault


I changed
doc.root.to_s.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

to
doc.root.to_a.join.gsub( /\<\/?#{artificial_root_tag}\>/, "" )

taking a hit on performance, but it allowed me to get through 100,000
iterations.
The memory usage though was stupid at 235Mb Real and 233Mb Virtual

The HTML I am parsing is a fairly small snippet

I'm running on Mac OsX.

Any ideas what and why?
thankyou

Daniel

I've managed to reproduce the error by changing to Benchmark.bmbm. It is

ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x3903a60; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x75b0750; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
ruby(3775) malloc: *** Deallocation of a pointer not malloced:
0x75c4950; This could be a double free(), or free() called with the
middle of an allocated block; Try setting environment variable
MallocHelp to see tools to help debug
/markup_mangler_benchmark.rb:79: [BUG] Segmentation fault