Writing long-running daemons without memory leaks?

[Toby DiPasquale]

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I've heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven't confirmed that) for the same reason. I understand that Ruby's GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I've read the Pickaxe's lone page on the subject (in the Duck
Typing chapter) and I've seen Why's article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html).

P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC...

 

[ara.t.howard]

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I've heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven't confirmed that) for the same reason. I understand that Ruby's GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I've read the Pickaxe's lone page on the subject (in the Duck
Typing chapter) and I've seen Why's article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html).

P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC...

i'm running dozens of ruby daemons, some of which are extremely (months at a
time) long lived and have not seen any gc issues. can you post a specific
(minimal) example that you find leaks memory?

regards.

-a

 

[ara.t.howard]

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

i cannot reproduce it:


harp:~ > cat a.rb
require 'socket'

class Buffer
def initialize size
@size = size
@buffer = []
@length = 0
end
attr_reader :length, :size
def full? ; @length == @size end
def empty? ; @length.zero? end
# Lets you fill up to capacity and then returns
# what's left over
def fill data
if full?
data
elsif @length + data.length < @size
@buffer << data
@length += data.length
nil
else
l = @size - @length
@buffer << data[0, l]
data[l..-1]
end
end
def expunge
buf = @buffer.join
@buffer.clear
@length = 0
buf
end
end # class Buffer

def read_long s
s.read( 4).unpack( "N")[0]
end

# main client handling thread logic
client_handler = lambda do |s|
begin
len = read_long s
buf = Buffer.new len
until buf.full?
x = s.read 4096
break if x.nil?
buf.fill x
end
puts "writing #{buf.length} bytes"
f = File.open "/dev/null", "w"
f.write buf.expunge
f.close
ensure
s.close
end
end

# main server loop
ss = TCPServer.new '127.0.0.1', 10001
begin
while true
s = ss.accept
puts "got connection"
Thread.start s, &client_handler
end
ensure
ss.close
end



harp:~ > cat b.rb
require 'socket'

def write_long s, l
s.write( [l].pack( "N"))
end

str = "a" * 65536

t = TCPSocket.new '127.0.0.1', 10001
write_long t, 1024 * str.length
1024.times { t.write str }
t.close



harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep 'ruby a.rb'|grep -v grep ;done
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ahoward 13671 12.7 23.1 239328 237956 pts/26 S 10:14 0:44 ruby a.rb
ahoward 13671 12.7 21.2 219404 218032 pts/26 S 10:14 0:45 ruby a.rb
ahoward 13671 12.8 22.3 230784 229396 pts/26 S 10:14 0:45 ruby a.rb
ahoward 13671 13.0 22.3 230784 229412 pts/26 S 10:14 0:45 ruby a.rb
ahoward 13671 13.1 22.3 230916 229416 pts/26 S 10:14 0:46 ruby a.rb
ahoward 13671 13.2 15.9 165364 163876 pts/26 S 10:14 0:46 ruby a.rb
ahoward 13671 13.3 22.3 230904 229424 pts/26 S 10:14 0:47 ruby a.rb
ahoward 13671 13.4 21.5 225116 221504 pts/26 R 10:14 0:47 ruby a.rb
ahoward 13671 13.5 28.7 296328 294940 pts/26 S 10:14 0:48 ruby a.rb
ahoward 13671 13.6 21.5 222480 221108 pts/26 S 10:14 0:48 ruby a.rb
ahoward 13671 13.7 22.3 230796 229400 pts/26 S 10:14 0:49 ruby a.rb
ahoward 13671 13.8 21.8 230804 224296 pts/26 R 10:14 0:49 ruby a.rb
ahoward 13671 13.9 27.3 296344 281256 pts/26 R 10:14 0:50 ruby a.rb
ahoward 13671 14.0 25.4 265672 261500 pts/26 R 10:14 0:50 ruby a.rb
ahoward 13671 14.1 22.3 230880 229384 pts/26 S 10:14 0:51 ruby a.rb
ahoward 13671 14.2 15.9 165340 163864 pts/26 S 10:14 0:51 ruby a.rb
ahoward 13671 14.3 20.8 230888 214376 pts/26 R 10:14 0:52 ruby a.rb
ahoward 13671 14.4 19.3 200220 198844 pts/26 S 10:14 0:52 ruby a.rb
ahoward 13671 14.5 21.2 230852 218484 pts/26 R 10:14 0:53 ruby a.rb
ahoward 13671 14.6 15.9 165312 163872 pts/26 R 10:14 0:53 ruby a.rb
ahoward 13671 14.8 28.1 290632 289256 pts/26 S 10:14 0:54 ruby a.rb
ahoward 13671 14.8 16.0 230776 164724 pts/26 R 10:14 0:54 ruby a.rb
ahoward 13671 14.9 21.6 230908 222412 pts/26 R 10:14 0:54 ruby a.rb
ahoward 13671 15.0 15.0 219356 154356 pts/26 R 10:14 0:55 ruby a.rb
ahoward 13671 15.1 16.2 230864 166464 pts/26 R 10:14 0:55 ruby a.rb
ahoward 13671 15.2 25.7 265764 264320 pts/26 S 10:14 0:56 ruby a.rb
ahoward 13671 15.4 22.3 230852 229388 pts/26 S 10:14 0:56 ruby a.rb
ahoward 13671 15.4 15.9 165312 163864 pts/26 S 10:14 0:57 ruby a.rb
ahoward 13671 15.5 15.9 165320 163880 pts/26 R 10:14 0:57 ruby a.rb
ahoward 13671 15.6 28.5 294368 292992 pts/26 S 10:14 0:58 ruby a.rb
ahoward 13671 15.7 22.3 230808 229388 pts/26 S 10:14 0:58 ruby a.rb
ahoward 13671 15.8 21.7 230816 223624 pts/26 R 10:14 0:59 ruby a.rb
ahoward 13671 15.9 15.9 165276 163872 pts/26 R 10:14 0:59 ruby a.rb
ahoward 13671 16.0 20.6 213384 212008 pts/26 S 10:14 0:59 ruby a.rb
ahoward 13671 16.1 22.3 230812 229396 pts/26 S 10:14 1:00 ruby a.rb
ahoward 13671 16.2 22.3 230812 229408 pts/26 S 10:14 1:00 ruby a.rb
ahoward 13671 16.2 15.9 165280 163872 pts/26 R 10:14 1:01 ruby a.rb
ahoward 13671 16.3 22.3 230812 229408 pts/26 S 10:14 1:01 ruby a.rb
ahoward 13671 16.4 19.3 200212 198836 pts/26 S 10:14 1:02 ruby a.rb
ahoward 13671 16.5 22.3 230840 229400 pts/26 S 10:14 1:02 ruby a.rb
ahoward 13671 16.5 15.4 159584 158204 pts/26 R 10:14 1:02 ruby a.rb
ahoward 13671 16.7 27.4 296424 281408 pts/26 R 10:14 1:03 ruby a.rb
ahoward 13671 16.8 19.3 203424 199044 pts/26 R 10:14 1:03 ruby a.rb
ahoward 13671 16.9 22.1 230888 227576 pts/26 R 10:14 1:04 ruby a.rb
ahoward 13671 17.0 25.7 265720 264108 pts/26 S 10:14 1:04 ruby a.rb
ahoward 13671 17.0 15.9 230812 163952 pts/26 R 10:14 1:05 ruby a.rb
ahoward 13671 17.1 27.1 296352 278484 pts/26 R 10:14 1:05 ruby a.rb
ahoward 13671 17.2 28.1 290640 289264 pts/26 S 10:14 1:06 ruby a.rb
ahoward 13671 17.3 15.9 165248 163860 pts/26 R 10:14 1:06 ruby a.rb
ahoward 13671 17.4 21.8 230920 224092 pts/26 R 10:14 1:07 ruby a.rb
ahoward 13671 17.5 22.0 228288 226912 pts/26 S 10:14 1:07 ruby a.rb
ahoward 13671 17.5 15.9 165264 163872 pts/26 R 10:14 1:07 ruby a.rb
ahoward 13671 17.6 22.0 228228 226852 pts/26 S 10:14 1:08 ruby a.rb
ahoward 13671 17.7 22.3 230868 229400 pts/26 S 10:14 1:08 ruby a.rb
ahoward 13671 17.8 14.8 153684 152304 pts/26 R 10:14 1:09 ruby a.rb
ahoward 13671 17.9 22.3 230812 229400 pts/26 S 10:14 1:09 ruby a.rb
ahoward 13671 18.0 21.7 225108 223732 pts/26 S 10:14 1:10 ruby a.rb
ahoward 13671 18.1 20.6 230924 212128 pts/26 R 10:14 1:10 ruby a.rb
ahoward 13671 18.1 14.4 149944 148564 pts/26 R 10:14 1:11 ruby a.rb
ahoward 13671 18.3 28.7 296332 294936 pts/26 S 10:14 1:11 ruby a.rb
ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby a.rb
ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby a.rb
ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby a.rb


so memory builds to about 25 and, presumably when the gc kicks in, drops to 15
- but it certainly stays in this range.

what os and ruby version?

-a

 

[Toby DiPasquale]

harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep [...]
ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby
a.rb
ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby
a.rb
ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby
a.rb


so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15
- but it certainly stays in this range.

what os and ruby version?

ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB
RAM. The other systems having this problem are all also running Linux
2.6.x with ruby 1.8.4.

One thing about the above: you do realize that's ~250MB and ~150MB, not
25 and 15, right?

 

[Booker C. Bense]

-----BEGIN PGP SIGNED MESSAGE-----

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

_ If your process is not using the memory, it may stay swapped
out, ( look at the real and virtual sizes ). If this is really
an issue, the standard way around it is to do the memory
intensive part in a forked subprocess that exits after it's
done.

_ Booker C. Bense


-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBRBsqMGTWTAjn5N/lAQHVbwP/XYqiyZsWzCqsYNfzP4rCe5cEzWbKBBfU
54A/x/DDnImzpM3MYcIG4Um2AAGFBYMrylxEeTv0Ot6RMi7Nl2NJLIzIg/6lGh1M
6d1ojbnddmUGfJXuYDZ2/lcWDWYzYCM6hYbWGh7h6eMYtNVZizE1k9ebA4zL+H0i
jFU8K8Aoe7k=
=ZbII
-----END PGP SIGNATURE-----