Process.waitFor() returns, but threads reading output/error streams are blocked on read

[Michael Grove]

java gurus,

i'm seeing some odd behavior running a process via Runtime.exec on redhat
linux (seen it on RHEL AS 2.1, 3, 4) with 1.4 and 1.5 JVMs (the only ones
i've tried). if i start a specific process via Runtime.exec and set up
threads to read in the out/err streams, i find that after Process.waitFor()
returns (~5 seconds) the threads are still stuck in InputStream.read(). i
would have expected them to return -1 (or fail with IOExceptions) almost
immediately after process termination. i've tried closing the streams from
the main thread and destroying the process, but that was not enough to break
the threads out of read. i found that if i changed my reader threads to
poll InputStream.available() they will find the streams closed after the
main thread closes them, although available() will return 0 indefinitely if
the main thread fails to close the streams manually.

i'll include the java source below, but the problem seems extremely specific
to the program i'm running in Runtime.exec(). i've been unable to duplicate
the problem with another program. the program is a non-interactive command
line tool (i'll call "cmd") that creates a socket connection to another
process on the same machine and requests an action to be performed (in my
case the action is "-start", passed in on the command line). the program
takes about 5 seconds to complete. upon successful completion no data is
written to standard out/err. the exit code is 0 on success, non-zero on
failure. i can confirm in the process list that the program does terminate
(which Process.waitFor() tells me anyway), and i can also confirm the
program performed the action i expected. i discovered that when my java
process is hung if i terminate the process the command line tool connects to
that will break the threads out of read() with a -1 response. that leads me
to believe that something was not cleaned up when the command line tool
exited. i've also found that if i invoke the command line tool
differently - e.g. i run "cmd -start" after a previous run of "cmd -start"
(close to a no-op) or run "cmd -status" - the java process terminates
normally. so something about the first call to "cmd -start" preventes
InputStream.read() from returning.

some questions:
* after Process.waitFor() returns, what could cause
Process.getInputStream().read() and Process.getErrorStream().read() to block
indefinitely? shouldn't they return -1 or receive IOExceptions almost
immediately?
* any thoughts on what this command line tool might be doing (perhaps not
closing its out/err streams gracefully when it terminates?) that would cause
these reads to block?

thanks.

-mike

import java.io.IOException;

public class cmdstart {
public static void main(String [] args) throws IOException,
InterruptedException {
String cmd = "cmd -start";
Process process = Runtime.getRuntime().exec(cmd);
MyReader outReader = new MyReader(process.getInputStream());
Thread outThread = new Thread(outReader);
outThread.start();
MyReader errReader = new MyReader(process.getErrorStream());
Thread errThread = new Thread(errReader);
errThread.start();
System.out.println("exit code is " + process.waitFor());
//process.destroy();
// process.getInputStream().close();
// process.getErrorStream().close();
outThread.join();
errThread.join();
System.out.println("joins completed");
}
}

import java.io.*;

public class MyReader implements Runnable {
private final InputStream is;

public MyReader(InputStream is) {
this.is = is;
}

public void run() {
// normally would read in while here, but these threads never get any
data -
// they are blocked on the first read() call forever.
int length;
try {
length = is.read();
if (length == -1) {
System.out.println("-1");
return;
}
if (length == 0) {
System.out.println("0");
return;
}
System.out.println("read something");
} catch (Throwable t) {
System.out.println(t);
}
}
}

 

[Thomas Hawtin]

* after Process.waitFor() returns, what could cause
Process.getInputStream().read() and Process.getErrorStream().read() to block
indefinitely? shouldn't they return -1 or receive IOExceptions almost
immediately?
* any thoughts on what this command line tool might be doing (perhaps not
closing its out/err streams gracefully when it terminates?) that would cause
these reads to block?

As a random guess forking (for instance executing another command)
and/or reopening stdout/stderr.

Apparently strace is the Linux command to use to see what a process is
doing. Never tried it myself.

Tom Hawtin

 

[Michael Grove]

thanks for the response.

you're right that something like fork occurs. i haven't fully tracked it
down via strace, but i remembered that the first time "cmd -start" is run it
actually bootstraps the other process it normally connects to if it finds it
not running. that process ends up started with init (pid 1) as it's parent
process. i didn't see exactly how this happened via strace, but it seemed
to occur right around a clone() call, which i see is pretty similar to fork.
i think the cloned process might actually be the one that bootstraps the
other process before it terminates. the cloned process vanishes before i
can get a chance to see what it's doing.

that should at least help me duplicate this problem with a "cmd" program of
my own creation. however i still wonder - is it correct behavior in java
that after the process has terminated the streams are not closed? even if
that answer is yes, i'm still confused why a polling InputStream.available()
would fail with an IOException once i closed the Process streams manually
yet InputStream.read() would remain blocked.

i know Runtime.exec() is a bit of a crapshoot and the javadocs indicate it
may not work well in a variety of cases. wondering though if this behavior
is 1) expected/correct or 2) a Runtime.exec() limitation or 3) a JVM bug.

thanks.

-mike

 

[John C. Bollinger]

you're right that something like fork occurs. i haven't fully tracked it
down via strace, but i remembered that the first time "cmd -start" is run it
actually bootstraps the other process it normally connects to if it finds it
not running. that process ends up started with init (pid 1) as it's parent
process. i didn't see exactly how this happened via strace, but it seemed
to occur right around a clone() call, which i see is pretty similar to fork.
i think the cloned process might actually be the one that bootstraps the
other process before it terminates. the cloned process vanishes before i
can get a chance to see what it's doing.

that should at least help me duplicate this problem with a "cmd" program of
my own creation. however i still wonder - is it correct behavior in java
that after the process has terminated the streams are not closed? even if
that answer is yes, i'm still confused why a polling InputStream.available()
would fail with an IOException once i closed the Process streams manually
yet InputStream.read() would remain blocked.

i know Runtime.exec() is a bit of a crapshoot and the javadocs indicate it
may not work well in a variety of cases. wondering though if this behavior
is 1) expected/correct or 2) a Runtime.exec() limitation or 3) a JVM bug.

My guess would be that the observed behavior is more-or-less correct, in
that the OS-level streams are not being closed, with the result that the
Java-level streams are not closed either. This could be the case if the
Process' stdin and stdout are inherited by the chain of other processes
that are forked off, and not closed by those processes I suspect that's
what is happening. If so, it's a minor bug in the bootstrapper,
possibly duplicated in the other process, and it can probably be
addressed in one or both of those places.

 

[Roedy Green]

try looking at java.util.prefs instead.

I meant java.lang.ProcessBuilder. I was thinking of somenone else's
question, and came back to this.

 

[Michael Grove]

thanks for clarifying - i couldn't imagine what preferences you had in mind
;-). program has to run on 1.4, so ProcessBuilder is not available.

i've asked the authors of the "cmd" program to make certain streams are
properly being closed, and will post any solution to the newsgroup.