Help: An error in Java program

[Sumedh]

Hi all,

So, im trying to run shell commands through Java using the
Runtime.getRuntime().exec() facility.

Now, basically I need to run the same command with different arguments
several times over, like thousands of times. So I have the exec() call
in a for loop.

Now after about some time I get this error:

java.io.IOException: Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:54)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at java.lang.Runtime.exec(Runtime.java:491)
at java.lang.Runtime.exec(Runtime.java:457)

Now, basically, knowing some unix, I know that ports opened by files
etc show up as file descriptors, and there must be some limit to how
many file descriptors can be opened at one time.

However, my shell command does not open any ports, and is basically a
perl script doing some parsing.

Also, after each command exec(), I do a process.waitFor(), so I ensure
that the process ends before I execute the loop again. (Assuming Java
does this correctly)

So, basically, is the communication between the Java Runtime and the
exec'd process opening so many ports/pipes between them? Im sure
there will be the basic input/output/error between the two programs.
Or, is the process not ending, and waitFor() not working, that several
processes are being exec'd at the same time?

Any ideas on what could be the problem? I've cross-posted to
comp.lang.java.programmers as well as comp.unix.questions, since im
not sure where the problem lies.

Thanks a lot for your answers/thoughts.

sumedh

 

[Gordon Beaton]

So, im trying to run shell commands through Java using the
Runtime.getRuntime().exec() facility.

Now, basically I need to run the same command with different
arguments several times over, like thousands of times. So I have the
exec() call in a for loop.

Now after about some time I get this error:

java.io.IOException: Too many open files

You need to close the streams connecting the Java application to the
child process.

Process p = Runtime.getRuntime().exec(cmd);

...

p.waitFor();
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();

/gordon

 

[Roedy Green]

Also, after each command exec(), I do a process.waitFor(), so I ensure
that the process ends before I execute the loop again. (Assuming Java
does this correctly)

Did you open streams to communicate with your spawned processes? if
so, did you remember to close them?

For other hints see http://mindprod.com/jgloss/exec.html

 

[Roedy Green]

p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();

Is that necessary even if you don't use them?

 

[Gordon Beaton]

Is that necessary even if you don't use them?

Using them has nothing to do with it - they're opened by exec() when
the child process is started.

/gordon

 

[joe]

So, im trying to run shell commands through Java using the
Runtime.getRuntime().exec() facility.

Now, basically I need to run the same command with different
arguments several times over, like thousands of times. So I have the
exec() call in a for loop.

Now after about some time I get this error:

java.io.IOException: Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:54)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at java.lang.Runtime.exec(Runtime.java:491)
at java.lang.Runtime.exec(Runtime.java:457)

Now, basically, knowing some unix, I know that ports opened by files
etc show up as file descriptors, and there must be some limit to how
many file descriptors can be opened at one time.

However, my shell command does not open any ports, and is basically a
perl script doing some parsing.

Also, after each command exec(), I do a process.waitFor(), so I
ensure that the process ends before I execute the loop
again. (Assuming Java does this correctly)

I don't know what process.waitfor() is supposed to do, but when you
spawn a child process, all its descriptors die with it. Unless Java is
doing something weird, it sounds like there's something else going on.

So, basically, is the communication between the Java Runtime and the
exec'd process opening so many ports/pipes between them? Im sure
there will be the basic input/output/error between the two programs.
Or, is the process not ending, and waitFor() not working, that
several processes are being exec'd at the same time?

If java opens pipes between the parent and child process, it should
close them when the child exits.

Have you tried running this under truss, or whatever system call
tracer your OS has? You can probably see what's happening that
way. You probably want to put some bracketing output lines to find the
right places though, since you'll likely get a ton of output.

Joe

 

[Roedy Green]

Using them has nothing to do with it - they're opened by exec() when
the child process is started.

I think you have discovered an other gotcha.

 

[Sumedh]

I think you have discovered an other gotcha.

Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
pipes opened between Java and the exec'd process. Apparently Java
(even with all its garbage collection) does not close those fd's even
after the process ends. That was causing the runaway fd's.

I explicitly closed the streams, and it works!

Thanks all for those who replied!!
Sumedh

 

[Steven Coco]

Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
pipes opened between Java and the exec'd process. Apparently Java
(even with all its garbage collection) does not close those fd's even
after the process ends. That was causing the runaway fd's.

I explicitly closed the streams, and it works!

Thanks all for those who replied!!
Sumedh

Can you post the core lines of your code--including the added close
commands?

--

.. Steven Coco .
.........................................................................
When you're not sure; "Confess your heart" says the Lord, "and you'll be
freed."

 

[Steven Coco]

I don't know what process.waitfor() is supposed to do, but when you
spawn a child process, all its descriptors die with it. Unless Java is
doing something weird, it sounds like there's something else going on.

That's what I was thinking.

--

.. Steven Coco .
.........................................................................
When you're not sure; "Confess your heart" says the Lord, "and you'll be
freed."

 

[Gordon Beaton]

Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
pipes opened between Java and the exec'd process. Apparently Java
(even with all its garbage collection) does not close those fd's
even after the process ends. That was causing the runaway fd's.

Garbage collection will *eventually* collect those streams, and the
descriptors will be closed when the appropriate finalizers run.

But gc is not triggered by lack of non-memory resources like file
descriptors, so when you invoke exec() in a loop you will use them up
long before gc gets involved.

The JVM can't close the descriptors automatically when the child
process terminates because there might be "grandchildren" that are
still active. The application itself needs to close them, and it
should probably wait for EOF before doing so.

/gordon