Reply to thread

Message: [QUOTE="Brad Volz, post: 4618769"] Thanks for the reply. I'm going to trim the original message a bit. Excellent. Thanks. Yes. It calls an external program to collect the actual data from the datastore. Having said that, I don't know that I can confidently say that the issue is in that particular block of code, as I can remove it and put in place something like: puts "hello !" and still experience the same problem. I'll try this again later with the new forkoff, but I currently have 0.0.1. Thanks for the test block. I ran it via - require 'rubygems' require 'forkoff' (0..255).forkoff do |f| ios = Array.new(8192){|i| IO.for_fd(i) rescue nil}.compact filenos = ios.map{|io| io.fileno} paths = ios.map{|io| io.path rescue nil} STDERR.puts "child : #{ Process.pid }" STDERR.puts "filenos : #{ filenos.inspect }" STDERR.puts "paths : #{ paths.inspect }" STDIN.gets end I won't past in the entire output, but here is the first and last child for comparison -- bradv:bvolz:$ ruby forkoff-test.rb child : 47036 filenos : [0, 1, 2, 4] paths : [nil, nil, nil, nil] child : 47288 filenos : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 255] paths : [nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil] /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb:42:in `pipe': Too many open files (Errno::EMFILE) from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 42:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 37:in `loop' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 37:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `initialize' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `new' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 32:in `times' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 32:in `forkoff' from forkoff-test.rb:5 bradv:bvolz:$ /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/ forkoff.rb:53:in `write': Broken pipe (Errno::EPIPE) from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 53:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 37:in `loop' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 37:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `initialize' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `new' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 34:in `forkoff' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 32:in `times' from /opt/local/lib/ruby/gems/1.8/gems/forkoff-0.0.1/lib/forkoff.rb: 32:in `forkoff' from forkoff-test.rb:5 If it's useful to know I am using ruby from macports on OS X 10.5 bradv:bvolz:$ ruby -v ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9] My current workaround is to use threadify instead of forkoff which is working beautifully. bradv:bvolz:$ ruby netflow.rb starting single process execution {:flows=>176915, :packets=>90580480, :octets=>81664177152} single exeution time: 138.792369842529 starting threadify execution {:flows=>176915, :packets=>90580480, :octets=>81664177152} threadify execution time: 85.1209449768066 cheers! Brad [/QUOTE]

Verification