Memory usage with blocks

[John Ky]

[Note: parts of this message were removed to make it a legal post.]

Hi all,

I've got some code, which when run uses a lot of memory (aprox 600 MB for a
particular given input). I am trying to figure out why it is using so much.

When I replace the following code:

head.play_back do |root|
block.call(root)
end

With this:

head.play_back(&block)

And make other similar changes. The program behaves exactly the same way,
but now uses 200MB less. Can someone explain to me why this is the case?

I use blocks everywhere and I'm worried, they're leaking in my code.

joky@linux08:~/wa/bvc-pure-cs/comet/src/swig_client_api/ruby $ ruby -v
ruby 1.8.7 (2008-06-20 patchlevel 22) [i686-linux]

Thanks

-John

 

[John Ky]

[Note: parts of this message were removed to make it a legal post.]

Hi,

I think it's the memory profiler I'm using:

MemoryProfiler.start

Nevermind.

Does anyone know how to properly find memory leaks in a ruby app?

Thanks,

-John

 

[Sebastian Hungerecker]

When I replace the following code:

=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 head.play_back do |root|
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0block.call(root)
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 end

With this:

=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 head.play_back(&block)

And make other similar changes. =A0The program behaves exactly the same w= ay,
but now uses 200MB less.

Consider the following:

def x(&block)
block
end

def a(&block)
tmp =3D create_a_two_hundred_mega_byte_object()
x {block.call} # Create a new block which closes over the current scope
end

def b(&block)
tmp =3D create_a_two_hundred_mega_byte_object()
x(&block) # Reuse the block that has been passed to this method.
# Do not create a new block
end

If I call b {puts "hello"}, the following will happen:
The block {puts "hello"} is created. This block holds a reference to any lo=
cal=20
variable that has been defined in the scope in which I call method b.
Now method b is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now x is called. x returns a Proc representing the block passed to b.
The method b ends, tmp goes out of scope and the 200mb object is garbage=20
collected.

If I call a {puts "hello"}, this happens:
The block {puts "hello"} is created. This block holds a reference to any lo=
cal=20
variable that has been defined in the scope in which I call method a.
Now method a is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now another block is created, which invokes the block passed to a.
This new block holds a reference to any local variable tmp defined in the=20
method body of a. Specifically it holds a reference to tmp.
Now x is called. x returns a Proc representing this new block.
The method a ends, but the 200mb is still referenced by the Proc that's=20
returned from a. So as long as reference to that Proc is stored, the 200mb=
=20
object will not be garbage collected.

I don't know whether that's what happens in your code, but it's certainly=20
worth looking into.

HTH,
Sebastian
=2D-=20
Jabber: (e-mail address removed)
ICQ: 205544826

 

[John Ky]

[Note: parts of this message were removed to make it a legal post.]

Hi Sebastian,

Why doesn't ruby detect that tmp isn't used in the block and not keep a
reference to it?

I found the offending code:

def play_back(&block)
$trace.method_block "PlaybackChain:lay_back(&)" do
sub_block = Proc.new do
@tail.play_back do |root|
block.call(root)
end
end
@block.call(sub_block)
end
end

def play_back(&block)
@heads.each do |head|
selector.select_each do |selection|
block.call(selection)
end
end
end

Replacing those to methods with

def play_back(&block)
$trace.method_block "PlaybackChain:lay_back(&)" do
sub_block = Proc.new do
@tail.play_back(&block)
end
@block.call(sub_block)
end
end

def play_back(&block)
@heads.each do |head|
head.play_back(&block)
end
end

wiped over 100 MB off the memory usage.

Thanks for your help,

-John

When I replace the following code:

head.play_back do |root|
block.call(root)
end

With this:

head.play_back(&block)

And make other similar changes. The program behaves exactly the same way,
but now uses 200MB less.

Consider the following:

def x(&block)
block
end

def a(&block)
tmp = create_a_two_hundred_mega_byte_object()
x {block.call} # Create a new block which closes over the current scope
end

def b(&block)
tmp = create_a_two_hundred_mega_byte_object()
x(&block) # Reuse the block that has been passed to this method.
# Do not create a new block
end

If I call b {puts "hello"}, the following will happen:
The block {puts "hello"} is created. This block holds a reference to any
local
variable that has been defined in the scope in which I call method b.
Now method b is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now x is called. x returns a Proc representing the block passed to b.
The method b ends, tmp goes out of scope and the 200mb object is garbage
collected.

If I call a {puts "hello"}, this happens:
The block {puts "hello"} is created. This block holds a reference to any
local
variable that has been defined in the scope in which I call method a.
Now method a is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now another block is created, which invokes the block passed to a.
This new block holds a reference to any local variable tmp defined in the
method body of a. Specifically it holds a reference to tmp.
Now x is called. x returns a Proc representing this new block.
The method a ends, but the 200mb is still referenced by the Proc that's
returned from a. So as long as reference to that Proc is stored, the 200mb
object will not be garbage collected.

I don't know whether that's what happens in your code, but it's certainly
worth looking into.

HTH,
Sebastian[/QUOTE]

 

[John Ky]

[Note: parts of this message were removed to make it a legal post.]

Runs heaps faster too. Woosh!

Hi Sebastian,

Why doesn't ruby detect that tmp isn't used in the block and not keep a
reference to it?

I found the offending code:

def play_back(&block)
$trace.method_block "PlaybackChain:lay_back(&)" do
sub_block = Proc.new do
@tail.play_back do |root|
block.call(root)
end
end
@block.call(sub_block)
end
end

def play_back(&block)
@heads.each do |head|
selector.select_each do |selection|
block.call(selection)
end
end
end

Replacing those to methods with

def play_back(&block)
$trace.method_block "PlaybackChain:lay_back(&)" do
sub_block = Proc.new do
@tail.play_back(&block)
end
@block.call(sub_block)
end
end

def play_back(&block)
@heads.each do |head|
head.play_back(&block)
end
end

wiped over 100 MB off the memory usage.

Thanks for your help,

-John




Consider the following:

def x(&block)
block
end

def a(&block)
tmp = create_a_two_hundred_mega_byte_object()
x {block.call} # Create a new block which closes over the current scope
end

def b(&block)
tmp = create_a_two_hundred_mega_byte_object()
x(&block) # Reuse the block that has been passed to this method.
# Do not create a new block
end

If I call b {puts "hello"}, the following will happen:
The block {puts "hello"} is created. This block holds a reference to any
local
variable that has been defined in the scope in which I call method b.
Now method b is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now x is called. x returns a Proc representing the block passed to b.
The method b ends, tmp goes out of scope and the 200mb object is garbage
collected.

If I call a {puts "hello"}, this happens:
The block {puts "hello"} is created. This block holds a reference to any
local
variable that has been defined in the scope in which I call method a.
Now method a is called with that block as an argument.
Then a 200mb object is created and stored in tmp.
Now another block is created, which invokes the block passed to a.
This new block holds a reference to any local variable tmp defined in the
method body of a. Specifically it holds a reference to tmp.
Now x is called. x returns a Proc representing this new block.
The method a ends, but the 200mb is still referenced by the Proc that's
returned from a. So as long as reference to that Proc is stored, the 200mb
object will not be garbage collected.

I don't know whether that's what happens in your code, but it's certainly
worth looking into.

HTH,
Sebastian

[/QUOTE]

 

[Sebastian Hungerecker]

Why doesn't ruby detect that tmp isn't used in the block and not keep a
reference to it?

Because tmp can still be accessed using the block even if it's not used in the
block directly:
def foo()
x = 5
lambda {puts "Hello world"}
end
hello_world_block = foo
eval("x", hello_world_block) #=> 5

HTH,
Sebastian