using lambda/Proc can prevent a lot of garbage collection

[Eric Mahurin]

Does anybody else think it is a serious issue that a Proc holds
references to all variables (including self) where it was
created (through a block)? These references are held through
Proc#binding. Although this does add some useful capability on
occasion, this causes excess memory to be used (worst case
could be a leaky program). Is this worth it? Personally, I
think not. I'd propose that Proc#binding return something that
only have access to variables referenced in the Proc. Even the
self from the defining context would be prohibited if not
referenced (possibly implicitly) in the Proc. Another option
would be to make these references weak so that they won't
prevent GC and those references would disappear if the
referenced object (including the variables/variable-table) is
GCed.

Comments?


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

 

[Kero]

Does anybody else think it is a serious issue that a Proc holds

references to all variables (including self) where it was
created (through a block)? These references are held through
Proc#binding. Although this does add some useful capability on
occasion, this causes excess memory to be used (worst case
could be a leaky program). Is this worth it? Personally, I
think not. I'd propose that Proc#binding return something that
only have access to variables referenced in the Proc. Even the
self from the defining context would be prohibited if not
referenced (possibly implicitly) in the Proc. Another option
would be to make these references weak so that they won't
prevent GC and those references would disappear if the
referenced object (including the variables/variable-table) is
GCed.

Which variables does a proc reference, when it uses eval, looks things
up via symbols, etc, etc?

You can't know in advance.

For an editor (or irb/completion) a best effort approach is OK, for a
binding, it is not OK.

+--- Kero ------------------------- kero@chello@nl ---+
| all the meaningless and empty words I spoke |
| Promises -- The Cranberries |
+--- M38c --- http://members.chello.nl/k.vangelder ---+

 

[Eric Mahurin]

=20
Which variables does a proc reference, when it uses eval,
looks things
up via symbols, etc, etc?
=20
You can't know in advance.
=20
For an editor (or irb/completion) a best effort approach is
OK, for a
binding, it is not OK.

Here is an example where a Proc can cause a memory leak:

ruby -e '
n=3D2**13;squares=3D(1..n).map{|i|a=3D(1..i).to_a;lambda{i*i}};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 169312 kB

You wouldn't expect each of these lambda's to need the local
"a", but the binding holds it. Right now, the programmer needs
to specifically think about this and clear these unexpected
object references by hand:

ruby -e ' n=3D2**13;squares =3D
(1..n).map{|i|a=3D(1..i).to_a;f=3Dlambda{i*i};a=3Dnil;f};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 10644 kB

To me, it seems kind of silly for the programmer to have to
worry about this level of detail. I'd expect most Proc's to
not need access to all variables in the defining context.

Here are the solutions I see:

1. Let the ruby programmer worry about it. They should assign
a variable to nil when a Proc has access to it and they are
done needing the variable.

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

4. Provide an additional facility for generating a Proc-like
object that doesn't have full variable access to the
surrounding context like a Proc has. This could be an Proc
method that generates a new Proc-like object, a Proc method
modifies self in-place, a different block syntax, or something
else.

Personally, I think #2 or #3 should be done, because it is
automatic, but this will could cause code that does eval within
the block or from Proc#binding to break - as Kero said.




=09
__________________________________=20
Yahoo! Music Unlimited=20
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/

 

[Gavin Kistner]

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

I believe that detecting references at compile time is tricky in a
dynamic language. Is there not a way in Ruby to retrieve a local
variable whose name is stored in another string? (Perhaps there isn't.

 

[David A. Black]

Hi --

Here is an example where a Proc can cause a memory leak:

ruby -e '
n=2**13;squares=(1..n).map{|i|a=(1..i).to_a;lambda{i*i}};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 169312 kB

You wouldn't expect each of these lambda's to need the local
"a", but the binding holds it. Right now, the programmer needs
to specifically think about this and clear these unexpected
object references by hand:

ruby -e ' n=2**13;squares =
(1..n).map{|i|a=(1..i).to_a;f=lambda{i*i};a=nil;f};
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
VmSize: 10644 kB

That's an interesting illustration, but I don't think I'd call it a
memory leak, since it's working as advertised. (At least, "memory
leak" to me implies something going wrong under the hood, so to
speak.)

To me, it seems kind of silly for the programmer to have to
worry about this level of detail. I'd expect most Proc's to
not need access to all variables in the defining context.

Here are the solutions I see:

1. Let the ruby programmer worry about it. They should assign
a variable to nil when a Proc has access to it and they are
done needing the variable.

2. Proc#binding should give a binding that only has variables
that the block accesses (determined when the compiled).

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

4. Provide an additional facility for generating a Proc-like
object that doesn't have full variable access to the
surrounding context like a Proc has. This could be an Proc
method that generates a new Proc-like object, a Proc method
modifies self in-place, a different block syntax, or something
else.

Nooooo...please... not *another* Proc/proc/lambda/block/method-like
object

Personally, I think #2 or #3 should be done, because it is
automatic, but this will could cause code that does eval within
the block or from Proc#binding to break - as Kero said.

It would be great if such things could be optimized away, but I just
can't reconcile doing it by "stingy" binding at the expense of the
dynamic techniques normally available. Maybe we all just have to be
aware of the fact that returning a closure means, in a sense, not
returning.


David

 

[Trans]

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

[1,2,3].each{|x| p x |}

The final '|' making it a closed block.

T.

 

[Trans]

Nooooo...please... not *another* Proc/proc/lambda/block/method-like
object

Why not? More the merrier! Still think they could be reduced as a
matter of difference in internal state of the same class rather than
wholly separate classes.

T.

 

[Eric Hodel]

Hi --



That's an interesting illustration, but I don't think I'd call it a
memory leak, since it's working as advertised. (At least, "memory
leak" to me implies something going wrong under the hood, so to
speak.)

A closure should only enclose the variables needed for its execution.
This is actually not very difficult but unfortunately eval('a')
prevents Ruby from excluding unbound variables in closures.

This is a (unfortunate IMO) feature of Ruby.

 

[Eric Hodel]

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

[1,2,3].each{|x| p x |}

The final '|' making it a closed block.

That won't make eval work in a closure like it should.

 

[Trans]

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

[1,2,3].each{|x| p x |}

The final '|' making it a closed block.

That won't make eval work in a closure like it should.

Like it *should*? I think eval works like it should, it's just that
having eval means you can't be sure of what you can exclude or not. (Or
is there something else?) The above explicity excludes all, essentially
no closure --that's the idea anyway. Once again a good notation proves
a problem.

T.

 

[Trans]

This seems essentially the same as the discussion on closed blocks.
Probably a bad idea to get rif of current behaivor, but there are good
reasons to add an additional for what you suggest. Of course, off the
bat there is a notation problem. My latest thought:

[1,2,3].each{|x| p x |}

The final '|' making it a closed block.

*This* will be really fun to parse.

Ugh, yea. Well, it sucked for other reasons anyway. Just tryng to churn
the pot.

T.

 

[Joel VanderWerf]

A closure should only enclose the variables needed for its execution.
This is actually not very difficult but unfortunately eval('a') prevents
Ruby from excluding unbound variables in closures.

This is a (unfortunate IMO) feature of Ruby.

Aren't there some useful hacks that depend on being able to get at vars
in the caller's scope (which need not be referenced in the block), using
the

eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's context,
using something like this:

def when_#{var} pattern=Object, &block
observer_map = @#{var}__observer_map ||= ObserverMap.new
if block
observer = eval "self", block

to find out what object is observing a variable. But that's not as
objectionable as getting access to arbitrary local vars in that context.

Does anyone remember--is there is anything that uses the latter kind of
access (arb. local vars) that is more than just a cute hack? Is there
anything really useful we would lose without that behavior?

 

[Logan Capaldo]

Eric Hodel wrote:



Aren't there some useful hacks that depend on being able to get at
vars
in the caller's scope (which need not be referenced in the block),
using
the

eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's context,
using something like this:

def when_#{var} pattern=Object, &block
observer_map = @#{var}__observer_map ||= ObserverMap.new
if block
observer = eval "self", block

to find out what object is observing a variable. But that's not as
objectionable as getting access to arbitrary local vars in that
context.

Does anyone remember--is there is anything that uses the latter
kind of
access (arb. local vars) that is more than just a cute hack? Is there
anything really useful we would lose without that behavior?

Speaking of eval, I've always wondered why ruby of all languages
needed a string based eval. What can you do with a string eval that
you can't do with something like:

eval(some_binding) { code }

some_binding of course would be optional just like currently.

with instance_variable_get() and instance_variable_get you
dynamically get at ivars. with send and/or instance eval you can
dynamically send messages. define_method blah blah... works just as
well as eval "def blah blah". This is something thats always kind of
irked me. And in my brave new world of evalless ruby, worse case
scenario you write a string to a temporary file and load it. But I
really don't believe thats necessary. Can anyone come up with a use
for an eval taking a string that CAN'T be acheieved by an eval that
takes a block? (well, irb I suppose). If nothing else I would
appreciate it greatly if eval could take a block in addition to a
string. I think this may have come off kind of like, so if I did I
apologize in advance ;-)

 

[Brian Mitchell]

Speaking of eval, I've always wondered why ruby of all languages
needed a string based eval. What can you do with a string eval that
you can't do with something like:

eval(some_binding) { code }

some_binding of course would be optional just like currently.

with instance_variable_get() and instance_variable_get you
dynamically get at ivars. with send and/or instance eval you can
dynamically send messages. define_method blah blah... works just as
well as eval "def blah blah". This is something thats always kind of
irked me. And in my brave new world of evalless ruby, worse case
scenario you write a string to a temporary file and load it. But I
really don't believe thats necessary. Can anyone come up with a use
for an eval taking a string that CAN'T be acheieved by an eval that
takes a block? (well, irb I suppose). If nothing else I would
appreciate it greatly if eval could take a block in addition to a
string. I think this may have come off kind of like, so if I did I
apologize in advance ;-)

Unfortunately, eval can do things that non-eval-code can't. Little
things like methods that take blocks, easier class definitions, quick
and dirty singleton definitions, etc... These _can_ be solved w/o eval
(well most of them) but it requires more code, more trickery, and/or
more discipline.

With that said, I am all for a Ruby w/o a direct need for eval besides
a REPL. Even a keyword-less way to do everything in Ruby would be nice
(singletons, I'm looking at you).

Brian.

 

[Eric Mahurin]

=20
I believe that detecting references at compile time is tricky
in a =20
dynamic language. Is there not a way in Ruby to retrieve a
local =20
variable whose name is stored in another string? (Perhaps
there isn't.

Compile-time meaning compile-time of the block - when the block
is created. I think of this as at compile-time, but with the
ruby interpreter it may be at run-time. I see no reason why it
shouldn't be able to determine what variables a block accesses
when it is created - with the exception of eval in the block
and eval using Proc#binding.


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

 

[Eric Mahurin]

Hi --
=20
On Sun, 9 Oct 2005, Eric Mahurin wrote:
=20
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
IO.readlines("/proc/#{Process.pid}/status").grep(/VmSize/).display'
=20
That's an interesting illustration, but I don't think I'd
call it a
memory leak, since it's working as advertised. (At least,
"memory
leak" to me implies something going wrong under the hood, so
to
speak.)

Well, C's malloc and free also work as advertised. And you can
easily make a C program that has a memory leak if you don't
pair these properly. You can blame it on the C
language/library or you can blame it on the C programmer, but
either way there is a memory leak when you don't pair these
properly. I see the above case as the same. Here is the
definition of "memory leak" I found on
http://www.webopedia.com/TERM/M/memory_leak.html :

"A bug in a program that prevents it from freeing up memory
that it no longer needs"

In the first example above that is what is going on. The
program is not freeing memory it no longer needs. Again, you
can blame this on the ruby programmer or the ruby langauge.=20
Since this is usually unwanted and/or unexpected, I choose to
blame the language.


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

 

[David A. Black]

Hi --

Well, C's malloc and free also work as advertised. And you can
easily make a C program that has a memory leak if you don't
pair these properly. You can blame it on the C
language/library or you can blame it on the C programmer, but
either way there is a memory leak when you don't pair these
properly. I see the above case as the same. Here is the
definition of "memory leak" I found on
http://www.webopedia.com/TERM/M/memory_leak.html :

"A bug in a program that prevents it from freeing up memory
that it no longer needs"

In the first example above that is what is going on. The
program is not freeing memory it no longer needs. Again, you
can blame this on the ruby programmer or the ruby langauge.
Since this is usually unwanted and/or unexpected, I choose to
blame the language.

The problem, though, is that the need (or lack thereof) can't be
determined, isn't it? It's not even safe to assume that the presence
of "eval" means "don't free the memory", nor that its absence means
not to. I mean, it's unlikely that someone would do:

alias :blah :eval
x = <some huge thing>
lambda { blah("x") }

but it's possible.


David

 

[Eric Mahurin]

Eric Hodel wrote:
=20
=20
Aren't there some useful hacks that depend on being able to
get at vars
in the caller's scope (which need not be referenced in the
block), using
the
=20
eval str, some_proc
=20
construct?
=20
I use it in my observable lib to get the self of the block's
context,
using something like this:
=20
def when_#{var} pattern=3DObject, &block
observer_map =3D @#{var}__observer_map ||=3D
ObserverMap.new
if block
observer =3D eval "self", block
=20
to find out what object is observing a variable. But that's
not as
objectionable as getting access to arbitrary local vars in
that context.

I assume you mean "eval str, some_binding" and 'eval "self",
block.binding'.

I've used this before too. This is kind of a way to get the
Binding#of_caller. I was thinking of this case when I made
this suggestion:

3. Proc#binding should hold weak references to variables that
the block doesn't access (at compile-time).

So in the above case, "self" would still be available
regardless of whether the block referenced self in the code
because there is no way it would be freed yet (it is in the
context of the caller) and Proc#binding would still have this
weak reference. "self" is a special case "variable", but it
should apply to more generic variables too.



=09
=09
__________________________________=20
Yahoo! Mail - PC Magazine Editors' Choice 2005=20
http://mail.yahoo.com

 

[Eric Mahurin]

=20


Aren't there some useful hacks that depend on being able to get at =20
vars
in the caller's scope (which need not be referenced in the block), =20
using
the

eval str, some_proc

construct?

I use it in my observable lib to get the self of the block's context,
using something like this:

def when_#{var} pattern=3DObject, &block
observer_map =3D @#{var}__observer_map ||=3D ObserverMap.new
if block
observer =3D eval "self", block

to find out what object is observing a variable. But that's not as
objectionable as getting access to arbitrary local vars in that =20
context.

Does anyone remember--is there is anything that uses the latter =20
kind of
access (arb. local vars) that is more than just a cute hack? Is there
anything really useful we would lose without that behavior?

--=20
vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

=20
Speaking of eval, I've always wondered why ruby of all
languages =20
needed a string based eval. What can you do with a string
eval that =20
you can't do with something like:
=20
eval(some_binding) { code }[/QUOTE]

I can't think immediately how giving a binding to this would be
useful, but I would like to see the block verision of eval:

class Kernel
def eval(*args)
if block_given?
yield
else
super
end
end
end

Right now, blocks can be used to create local variables (as
long as they don't have the same name as an existing variable).
You could use this eval to do this easily. This could change
with Ruby 2 though. Maybe there are other uses.

some_binding of course would be optional just like currently.
=20
with instance_variable_get() and instance_variable_get you =20
dynamically get at ivars. with send and/or instance eval you
can =20
dynamically send messages. define_method blah blah... works
just as =20
well as eval "def blah blah". This is something thats always
kind of =20
irked me. And in my brave new world of evalless ruby, worse
case =20
scenario you write a string to a temporary file and load it.
But I =20
really don't believe thats necessary. Can anyone come up with
a use =20
for an eval taking a string that CAN'T be acheieved by an
eval that =20
takes a block? (well, irb I suppose). If nothing else I would
=20
appreciate it greatly if eval could take a block in addition
to a =20
string. I think this may have come off kind of like, so if I
did I =20
apologize in advance ;-)

I looked at this a while back and found that most cases eval is
used, it is used to create a method on the fly. You could
probably find a way to use define_method instead of eval("def
...") for most of those cases. But, you'd still have the
problem of this thread - a Proc can prevent garbage collection.
Using define_method with a block will give that block access
to all of the variable in the defining context. And that
block/Proc will stick around indefinitely in the method that
you are defining.

Assuming the issue of this thread is solved, you still have
another reason why eval("def ...") may be used over
define_method. def creates a completely local variable
context. You don't have that with define_method. You still
have to worry about the variables in the block not interacting
with variables in its context. This brings us back to another
thread where I was talking about adding some local variable
facilities.

Currently, I'm using eval("def ...") in a way that
define_method can't compete. I'm using it to make macro-like
methods - flattening many levels of method calls into one
method. Granted this is a performance reason, but
define_method can't do it.


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around=20
http://mail.yahoo.com=20

 

[Trans]

really don't believe thats necessary. Can anyone come up with a use
for an eval taking a string that CAN'T be acheieved by an eval that
takes a block? (well, irb I suppose).

No closure.

T.