Process does not receive EOF when reading from Process.getOutputStream()

[Reid Madsen]

I'm using Runtime.exec() to run a command via xargs so that I can pass
a long list of arguments (too long to put on the command line). 99% of
the time this works perfectly. In the 1% case that fails, the xargs process
hangs reading from its input stream. It acts as if it has not detected that
we have closed its input stream in the java application.

I'm running the JDK 1.4.1_03 version on Solaris 2.9 with 24 CPU's.

Here are the basic elements of this implementation:

1) I have created a subprocess using:

Process xargs = Runtime.exec("/bin/xargs cleartool lsview -long");

BufferedReader stdOut =
new BufferedReader(new InputStreamReader(xargs.getInputStream()));
BufferedReader stdErr =
new BufferedReader(new InputStreamReader(xargs.getErrorStream()));
PrintStream stdIn =
new PrintStream(new BufferedOutputStream(xargs.getOutputStream()), true);

2) I have then created two threads, one to dispose of stderr from xargs,
and another to feed input into xargs:

new Thread() {
public void run() {
// Discard all output on stderr
while (stdErr.readln() != null);
}
}.start();

// The list of arguments to pass to the process via /bin/xargs
String[] args= <Array of strings>;

new Thread() {
public void run() {
// Feed arguments to process stdin
for (int i=0; i<args.length; ++i) {
stdIn.println(args);
// close process stdin to signal end of input
// process should then process arguments and exit
stdIn.flush();
stdIn.close();
}
}.start();

3) Then I process the output from process stdout directly:

String input;
while ((input = stdOut.readln()) != null) {
// process input
}

4) Finally I wait for the process to exit:

int exitStatus = xargs.waitFor();

This seems to be the correct way of doing this, and it works
most of the time. Has anyone experienced this problem before?
If so, what is the solution? Or, can you give me some ideas of where to look?
Thank you in advance for any help you may offer.

Reid Madsen
i2 Technologies

 

[Sudsy]

Reid Madsen wrote:

2) I have then created two threads, one to dispose of stderr from xargs,
and another to feed input into xargs:

So where's the one which gobbles stdout?

 

[Reid Madsen]

Sudsy,
The stdout from the process is read by the loop in #3.
All three streams are being handled -- stdin and stderr by background
threads, and stdin by the current thread.
Reid

 

[Missaka Wijekoon]

What is xargs doing when it is hung?

Do a:
truss -p <pid_of_xargs>

What system call is it in? It it stuck in a read or write on
filedescriptor 0/1/2?

What does cleartool do? What does a truss of cleartool say its doing?

-Misk

I'm using Runtime.exec() to run a command via xargs so that I can pass
a long list of arguments (too long to put on the command line). 99% of
the time this works perfectly. In the 1% case that fails, the xargs process
hangs reading from its input stream. It acts as if it has not detected that
we have closed its input stream in the java application.

I'm running the JDK 1.4.1_03 version on Solaris 2.9 with 24 CPU's.

Here are the basic elements of this implementation:

1) I have created a subprocess using:

Process xargs = Runtime.exec("/bin/xargs cleartool lsview -long");

BufferedReader stdOut =
new BufferedReader(new InputStreamReader(xargs.getInputStream()));
BufferedReader stdErr =
new BufferedReader(new InputStreamReader(xargs.getErrorStream()));
PrintStream stdIn =
new PrintStream(new BufferedOutputStream(xargs.getOutputStream()), true);

2) I have then created two threads, one to dispose of stderr from xargs,
and another to feed input into xargs:

new Thread() {
public void run() {
// Discard all output on stderr
while (stdErr.readln() != null);
}
}.start();

// The list of arguments to pass to the process via /bin/xargs
String[] args= <Array of strings>;

new Thread() {
public void run() {
// Feed arguments to process stdin
for (int i=0; i<args.length; ++i) {
stdIn.println(args);
// close process stdin to signal end of input
// process should then process arguments and exit
stdIn.flush();
stdIn.close();
}
}.start();

3) Then I process the output from process stdout directly:

String input;
while ((input = stdOut.readln()) != null) {
// process input
}

4) Finally I wait for the process to exit:

int exitStatus = xargs.waitFor();

This seems to be the correct way of doing this, and it works
most of the time. Has anyone experienced this problem before?
If so, what is the solution? Or, can you give me some ideas of where to look?
Thank you in advance for any help you may offer.

Reid Madsen
i2 Technologies

--
========================================================================
Missaka Wijekoon (a.k.a. Misk) (e-mail address removed)
Sr. Software Engineer http://www.villageEdocs.com
VillageEdocs
========================================================================

 

[Reid Madsen]

Missaka,

I should have included that earlier -- thanks for asking.
Running truss on the xargs pid shows that it is hung in a read on file
descriptor 0.
The cleartool command isn't running, it has not been launched yet -- or it
has
been launched, finished, and now xargs is trying to read more arguments.

Remember the parent has closed the xargs input stream -- xargs should have
seen
an EOF and exited.

FYI, cleartool is part of Rational/IBM Clearcase -- a source code control
system.
I should also note that I've only seen this occur with xargs -- so far.
Maybe it's a
bug in the xargs implementation.

Thanks again.
Reid Madsen
i2 Technologies

What is xargs doing when it is hung?

Do a:
truss -p <pid_of_xargs>

What system call is it in? It it stuck in a read or write on
filedescriptor 0/1/2?

What does cleartool do? What does a truss of cleartool say its doing?

-Misk

I'm using Runtime.exec() to run a command via xargs so that I can pass
a long list of arguments (too long to put on the command line). 99% of
the time this works perfectly. In the 1% case that fails, the xargs process
hangs reading from its input stream. It acts as if it has not detected that
we have closed its input stream in the java application.

I'm running the JDK 1.4.1_03 version on Solaris 2.9 with 24 CPU's.

Here are the basic elements of this implementation:

1) I have created a subprocess using:

Process xargs = Runtime.exec("/bin/xargs cleartool lsview -long");

BufferedReader stdOut =
new BufferedReader(new InputStreamReader(xargs.getInputStream()));
BufferedReader stdErr =
new BufferedReader(new InputStreamReader(xargs.getErrorStream()));
PrintStream stdIn =
new PrintStream(new BufferedOutputStream(xargs.getOutputStream()), true);

2) I have then created two threads, one to dispose of stderr from xargs,
and another to feed input into xargs:

new Thread() {
public void run() {
// Discard all output on stderr
while (stdErr.readln() != null);
}
}.start();

// The list of arguments to pass to the process via /bin/xargs
String[] args= <Array of strings>;

new Thread() {
public void run() {
// Feed arguments to process stdin
for (int i=0; i<args.length; ++i) {
stdIn.println(args);
// close process stdin to signal end of input
// process should then process arguments and exit
stdIn.flush();
stdIn.close();
}
}.start();

3) Then I process the output from process stdout directly:

String input;
while ((input = stdOut.readln()) != null) {
// process input
}

4) Finally I wait for the process to exit:

int exitStatus = xargs.waitFor();

This seems to be the correct way of doing this, and it works
most of the time. Has anyone experienced this problem before?
If so, what is the solution? Or, can you give me some ideas of where to look?
Thank you in advance for any help you may offer.

Reid Madsen
i2 Technologies

--
========================================================================
Missaka Wijekoon (a.k.a. Misk) (e-mail address removed)
Sr. Software Engineer http://www.villageEdocs.com
VillageEdocs
========================================================================

 

[Andrew Thompson]

I should have included that earlier ...

(pssst..)
...What you might have left out 'now'
was (most of) the 95 lines of earlier
post that was no longer needed.

Otherwise you might upset Roedy.
(shhhh..) ;-)

 

[Missaka Wijekoon]

Have you tried closing stdOut and stdErr at the end of their respective
threads? The reason I do this is that I when I perform this operation
in C, I do i/o multiplexing with poll()...and if you aren't careful
about reading everthing and closing everything, somethimes the buffering
gets in the way.

-Misk

Missaka,

I should have included that earlier -- thanks for asking.
Running truss on the xargs pid shows that it is hung in a read on file
descriptor 0.
The cleartool command isn't running, it has not been launched yet -- or it
has
been launched, finished, and now xargs is trying to read more arguments.

Remember the parent has closed the xargs input stream -- xargs should have
seen
an EOF and exited.

FYI, cleartool is part of Rational/IBM Clearcase -- a source code control
system.
I should also note that I've only seen this occur with xargs -- so far.
Maybe it's a
bug in the xargs implementation.

Thanks again.
Reid Madsen
i2 Technologies

What is xargs doing when it is hung?

Do a:
truss -p <pid_of_xargs>

What system call is it in? It it stuck in a read or write on
filedescriptor 0/1/2?

What does cleartool do? What does a truss of cleartool say its doing?

-Misk

I'm using Runtime.exec() to run a command via xargs so that I can pass
a long list of arguments (too long to put on the command line). 99% of
the time this works perfectly. In the 1% case that fails, the xargs

process

hangs reading from its input stream. It acts as if it has not detected

that

we have closed its input stream in the java application.

I'm running the JDK 1.4.1_03 version on Solaris 2.9 with 24 CPU's.

Here are the basic elements of this implementation:

1) I have created a subprocess using:

Process xargs = Runtime.exec("/bin/xargs cleartool lsview -long");

BufferedReader stdOut =
new BufferedReader(new InputStreamReader(xargs.getInputStream()));
BufferedReader stdErr =
new BufferedReader(new InputStreamReader(xargs.getErrorStream()));
PrintStream stdIn =
new PrintStream(new BufferedOutputStream(xargs.getOutputStream()),

true);

2) I have then created two threads, one to dispose of stderr from xargs,
and another to feed input into xargs:

new Thread() {
public void run() {
// Discard all output on stderr
while (stdErr.readln() != null);
}
}.start();

// The list of arguments to pass to the process via /bin/xargs
String[] args= <Array of strings>;

new Thread() {
public void run() {
// Feed arguments to process stdin
for (int i=0; i<args.length; ++i) {
stdIn.println(args);
// close process stdin to signal end of input
// process should then process arguments and exit
stdIn.flush();
stdIn.close();
}
}.start();

3) Then I process the output from process stdout directly:

String input;
while ((input = stdOut.readln()) != null) {
// process input
}

4) Finally I wait for the process to exit:

int exitStatus = xargs.waitFor();

This seems to be the correct way of doing this, and it works
most of the time. Has anyone experienced this problem before?
If so, what is the solution? Or, can you give me some ideas of where to

look?

Thank you in advance for any help you may offer.

Reid Madsen
i2 Technologies

--
========================================================================
Missaka Wijekoon (a.k.a. Misk) (e-mail address removed)
Sr. Software Engineer http://www.villageEdocs.com
VillageEdocs
========================================================================

--
========================================================================
Missaka Wijekoon (a.k.a. Misk) (e-mail address removed)
Sr. Software Engineer http://www.villageEdocs.com
VillageEdocs
========================================================================

 

[Marc Slemko]

I'm using Runtime.exec() to run a command via xargs so that I can pass
a long list of arguments (too long to put on the command line). 99% of
the time this works perfectly. In the 1% case that fails, the xargs process
hangs reading from its input stream. It acts as if it has not detected that
we have closed its input stream in the java application.

I'm running the JDK 1.4.1_03 version on Solaris 2.9 with 24 CPU's.

I haven't taken the time to look at your example much, but have seen
issues that sound similar before in 1.4.1_03. See this bug:

http://developer.java.sun.com/developer/bugParade/bugs/4843136.html

Synopsis (process) pipe file descriptor from Runtime.exec not being closed

it was fixed in 1.4.1_05.

 

[Reid Madsen]

Thank you Marc!!

When I run into bugs like this I begin to question my design.
Is the design flawed? Or, is the bug elsewhere.

Your timely posting validates my initial premise -- that the closing of
stdin was
not being propagated all the way down to the process.

Thanks again.

Reid Madsen
i2 Technologies