G
Geert Fannes
Hello,
I noticed that ruby's disc performance drops drastically when a large
array is allocated. I think it has to do with garbage collection since
the performance increases again by disabling the garbage collection. I
created a small test program to illustrate the problem:
#
#begin of program
#
allocateBefore=true
useFileLoop=true
disableGC=false
GC.disable if disableGC
#create a file containing 100000 lines of 'test'
File.open('testfile','w'){|fo| 100000.times{fo.puts 'test'}}
largeArray=Array.new(20000000) if allocateBefore
if useFileLoop
File.open('testfile') do |fi|
fi.each{|line|}
end
else
1000000.times{|i|}
end
largeArray=Array.new(20000000) if !allocateBefore
#
#end of program
#
On my home pc, the above program takes 3.225 sec. If I allocate the
large array AFTER the fi.each-loop by setting allocateBefore=false, it
takes only 0.467 sec. The same good performance occurs when I disable
the garbage collection by setting disableGC=true. Unfortunately,
disabling GC is not an option in my real application since my file is
a lot larger and all my memory gets consumed very fast.
If I play with the allocateBefore and disableGC when the
1000000.times-loop is enabled (by setting useFileLoop=false), I don't
get this difference anymore.
Any idea what is going on here? How can I achieve a good file
performance with large arrays in memory?
Greets,
Geert Fannes.
I noticed that ruby's disc performance drops drastically when a large
array is allocated. I think it has to do with garbage collection since
the performance increases again by disabling the garbage collection. I
created a small test program to illustrate the problem:
#
#begin of program
#
allocateBefore=true
useFileLoop=true
disableGC=false
GC.disable if disableGC
#create a file containing 100000 lines of 'test'
File.open('testfile','w'){|fo| 100000.times{fo.puts 'test'}}
largeArray=Array.new(20000000) if allocateBefore
if useFileLoop
File.open('testfile') do |fi|
fi.each{|line|}
end
else
1000000.times{|i|}
end
largeArray=Array.new(20000000) if !allocateBefore
#
#end of program
#
On my home pc, the above program takes 3.225 sec. If I allocate the
large array AFTER the fi.each-loop by setting allocateBefore=false, it
takes only 0.467 sec. The same good performance occurs when I disable
the garbage collection by setting disableGC=true. Unfortunately,
disabling GC is not an option in my real application since my file is
a lot larger and all my memory gets consumed very fast.
If I play with the allocateBefore and disableGC when the
1000000.times-loop is enabled (by setting useFileLoop=false), I don't
get this difference anymore.
Any idea what is going on here? How can I achieve a good file
performance with large arrays in memory?
Greets,
Geert Fannes.