Too many open files (Errno::EMFILE) on Windows

[Jim Surname]

It seems that a number of people have run into this problem in different
situations. In my case, I have a large form in my Rails app with file
uploads (enctype="multipart/form-data").

When the form is submitted, the read_multipart method in CGI creates a
Tempfile for every element until the maximum number of open files is
reached and the process deadlocks.

The attached script will try to open new files forever. On my machine,
before updating Ruby, the script dies when trying to open the 510th
file.

On most platforms (not Windows), increasing the open file limit is
simple. The problem lies in the C Run-Time libs. This link explains the
situation better than I can:
http://msdn2.microsoft.com/en-us/library/6e3b887c(VS.71).aspx

I was able to increase the limit by adding a call to _setmaxstdio in the
Ruby main.c file and then compiling everything from source. Of course, I
can forsee my form growing in size to more than 2048 elements.

The ultimate fix will be to use the Win32 API or MFC for dealing with
files on Windows instead of the C Run-Time API, which should up the
limit to some insanely high number.

Attachments:
http://www.ruby-forum.com/attachment/1348/too_many_open_files.rb

 

[Ryan Davis]

The ultimate fix will be to use the Win32 API or MFC for dealing with
files on Windows instead of the C Run-Time API, which should up the
limit to some insanely high number.

no. the ultimate fix is to use block form:

File.open path, mode do |f|
# ...
end

 

[Daniel Berger]

It seems that a number of people have run into this problem in different
situations. In my case, I have a large form in my Rails app with file
uploads (enctype=3D"multipart/form-data").

When the form is submitted, the read_multipart method in CGI creates a
Tempfile for every element until the maximum number of open files is
reached and the process deadlocks.

The attached script will try to open new files forever. On my machine,
before updating Ruby, the script dies when trying to open the 510th
file.

<snip>

# Code snippet in question
count =3D 1
files =3D Array.new
loop do
print count.to_s + "\n"
files << File.new("test#{count}.tmp", "w")
count +=3D 1
end

What you're doing here is pushing the file _descriptor_ onto the
'files' array, not the file name, and you're never closing the
descriptor. Simply increasing the descriptor limit isn't going to cut
it.

I'm not exactly sure what you're trying to accomplish, since I don't
know if you need to write to those files at some later point or what,
but you'll need to change your approach.

Also, the 'tempfile' library may be of some interest to you. It's part
of the standard library.

Regards,

Dan

 

[Jim Surname]

What you're doing here is pushing the file _descriptor_ onto the

'files' array, not the file name, and you're never closing the
descriptor. Simply increasing the descriptor limit isn't going to cut
it.

Perhaps I was unclear as to what the problem is. The example code I gave
you just points out a problem with the C run-Time API on Windows which
is used by Ruby for handling files. The example code demonstrates how
this limit affects Ruby programs. Your responses focus just on my
example code, completely ignoring the rest of my message.

My hope was that someone could help clarify where the actual problem is
and either suggest how I might work around this problem or how I might
patch Ruby to fix it. I know for damn sure that MS will not fix the
problem in their C run-time; otherwise, why would anyone use their
proprietary APIs? So the only way to fix it is by patching Ruby.

In my Ruby installation folder there is a file: lib/ruby/1.8/cgi.rb
In that file is a function: read_multipart
In read_multipart there is a block of code that looks like this:
---
loop do
head = nil
if 10240 < content_length
require "tempfile"
body = Tempfile.new("CGI")
else
begin
require "stringio"
body = StringIO.new
rescue LoadError
require "tempfile"
body = Tempfile.new("CGI")
end
end
---

In my Rails application, I have a multipart form with about ten billion
fields, including some file upload fields. When this form is submitted,
the POST data gets passed to this read_multipart method in some way. The
read_multipart method seems to be creating a Tempfile for every field
passed from the form. When the limit for file descriptors is reached,
the Ruby process deadlocks and I have to kill and restart it.

My temporary "fix" was to increase the limit to it's maximum which, on
Windows, requires a recompile of Ruby. I already know that this will not
be sufficient, otherwise I would not be posting here.

There may be a better way to write the read_multipart method so that it
doesn't need so many Tempfiles. That would probably be easier than
updating Ruby's File class to use something other than the standard C
API on Windows.