newbie question regarding memory leak

[Venkat Bagam]

Hi, Recently, I have written a ruby program for web-scraping. But its a
bit slow and consumes enormous memory due to a possible memory leak. I
am very new to these memory leak issues. Please find my code in the
attachment.

Can anyone patiently go through my code and figure out where the
possible mem leak occurs? Any help/suggestion greatly appreciated...

thanks & regards,
Venkat Bagam

Attachments:
http://www.ruby-forum.com/attachment/1561/test4.rb

 

[Benjamin Oakes]

Hi, Recently, I have written a ruby program for web-scraping. But its a
bit slow and consumes enormous memory due to a possible memory leak. I
am very new to these memory leak issues. Please find my code in the
attachment.

Can anyone patiently go through my code and figure out where the
possible mem leak occurs? Any help/suggestion greatly appreciated...

thanks & regards,
Venkat Bagam

Attachments:
http://www.ruby-forum.com/attachment/1561/test4.rb

Where do you think your memory leak is? (Also, why don't you think
Ruby's garbage collecting is handling it? Maybe you have references
to things you're not using anymore?)

-- Ben (aka divokz)

 

[Venkat Bagam]

Where do you think your memory leak is? (Also, why don't you think
Ruby's garbage collecting is handling it? Maybe you have references
to things you're not using anymore?)

I have assigned the useless object references to nil. You can observe, I
assigned

page = nil
page_links = nil
doc = nil
file_name = nil at some point of the program, where I no longer needed
them. Doesn't assigning a ref to nil, completely free its object memory?
Isn't nil an empty object? Does nil also claims memory? I am very sorry
if my questions doesn't make any sense !!!

 

[Benjamin Oakes]

I have assigned the useless object references to nil. You can observe, I
assigned

page = nil
page_links = nil
doc = nil
file_name = nil at some point of the program, where I no longer needed
them. Doesn't assigning a ref to nil, completely free its object memory?
Isn't nil an empty object? Does nil also claims memory? I am very sorry
if my questions doesn't make any sense !!!

I haven't had much time to review the code you've posted, but
regarding your questions here, nil *does* take up memory, however,
there is only *one* instance of nil (or NilClass, rather), if I
remember correctly. Whenever you use "nil," you are using the
reference to that class. (That is, every nil points to the same
place.) If you believe that your objects are still resident after
garbage collecting, there's a chance that you still have references to
them somewhere in memory. (For example, you may set a variable that
once held an array to nil, but then if that array had references to
other objects, there would still be a reference count above 0, so the
objects would stick around. I'm not sure if that example would hold
true, but I believe that's what the general idea is.)

I hope this helps,

-- Ben

 

[ruby rails]

Atlast I figured it out. Rather than declaring @agent as an instance
variable, i tried declaring it as a local variable and that does the
trick. Previously it used to eat up 400-500 MB of memory but now, it
uses a max of 15 MB.

I modified my code from

def initialize
@agent = WWW::Mechanize.new
end

def fetch_links(start, finish, count)
page = WWW::Mechanize:age
........
........
end

to look like below

def initialize

end

def fetch_links(start, finish, count)
agent = WWW::Mechanize.new
page = WWW::Mechanize:age
end

Any interpretations from this?