Problem with InputStream & Process

[none]

Hello *,

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

Here is my code:
1 public class Main {
2 public static void main(String[] args) {
3 ProcessBuilder pb = new ProcessBuilder("/tmp/test.py");
4
5 try {
6 Process p = pb.start();
7 p = pb.start();
8
9 BufferedReader output = new BufferedReader(
10 new InputStreamReader(p.getInputStream()));
11 String line;
12
13 while ((line = output.readLine()) != null) {
14 System.out.println(line);
15 }
16
17 p.waitFor();
18 }
19 catch(Exception ex) {
20 ex.printStackTrace();
21 }
22 }
23}

In line 13 my application blocks til process termination of test.py and
afterwards I get all outputs of test.py and miss the intermediate steps...

Here my test program:
test.py:
1 #!/usr/bin/python
2 import time
3 cnt = 5
4 while cnt:
5 time.sleep(1)
6 print "progress"
7 cnt-=1

Every 1 second I should get "progress" into my application but it does
not seem do work.
I played around with streaming operations and Threads but without success.

Any help are welcome!

java version: Sun-Java 1.6.0u13 on Linux/i386
(same issue also on Open JDK Java 1.6.0_0-b12)

Nice greetings,
Harald

--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[Harald Krammer]

Hello *,

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

Here is my code:
1 public class Main {
2 public static void main(String[] args) {
3 ProcessBuilder pb = new ProcessBuilder("/tmp/test.py");
4
5 try {
6 Process p = pb.start();
7 p = pb.start();
8
9 BufferedReader output = new BufferedReader(
10 new InputStreamReader(p.getInputStream()));
11 String line;
12
13 while ((line = output.readLine()) != null) {
14 System.out.println(line);
15 }
16
17 p.waitFor();
18 }
19 catch(Exception ex) {
20 ex.printStackTrace();
21 }
22 }
23}

In line 13 my application blocks til process termination of test.py and
afterwards I get all outputs of test.py and miss the intermediate steps...

Here my test program:
test.py:
1 #!/usr/bin/python
2 import time
3 cnt = 5
4 while cnt:
5 time.sleep(1)
6 print "progress"
7 cnt-=1

Every 1 second I should get "progress" into my application but it does
not seem do work.
I played around with streaming operations and Threads but without success.

Any help are welcome!

java version: Sun-Java 1.6.0u13 on Linux/i386
(same issue also on Open JDK Java 1.6.0_0-b12)

Nice greetings,
Harald

Sorry for bad sender address. I forgot to configure my news client.
--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[Harald Krammer]

I noticed a couple of issues with your code. You start your process twice
(lines 6 & 7). I doubt this would cause your problem, however.




You also don't read stderr. Although doing so is likely not to affect your
program, it's a Process "gotcha" which can cause Java and your external prog
to deadlock. The solution (which has been discussed in this forum many
times--Google) is either to combine stdout and stderr or to simultaneously
read both.

Thx for the hint. In my real life application I added
'pb.redirectErrorStream(true);' to merge stdin and stderr into one
stream, so a deadlock should be avoided.

The blocking behavior may be caused by a mismatch between what
BufferedReader expects as line terminator and what Python produces. Does
Python print include a newline? When this code finally does get input, are
the values on separate lines or all in one string? If they're on separate
lines, the Python output buffer is not flushing.

Yes, python print includes a line terminator.
I verified it with strace and got following trace:
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
write(1, "hello\n", 6) = 6

Every values has it own line so it is separated. In my test case the
while loop runs 5 times. If the python app runs alone, all looks fine,
because every second I get an item on the terminal.

I found out when the output of the external application exceed more than
64k then I get data from the stream. Hmmmm... It looks like here is a
"hidden" 64k buffer.



--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[Arne Vajhøj]

none schrieb: ....
Sorry for bad sender address. I forgot to configure my news client.

Not a big problem.

Fake email addresses to avoid spam and even fake identities are
often seen.

Arne

 

[Arne Vajhøj]

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

Here is my code:
1 public class Main {
2 public static void main(String[] args) {
3 ProcessBuilder pb = new ProcessBuilder("/tmp/test.py");
4
5 try {
6 Process p = pb.start();
7 p = pb.start();
8
9 BufferedReader output = new BufferedReader(
10 new InputStreamReader(p.getInputStream()));
11 String line;
12
13 while ((line = output.readLine()) != null) {
14 System.out.println(line);
15 }
16
17 p.waitFor();
18 }
19 catch(Exception ex) {
20 ex.printStackTrace();
21 }
22 }
23}

In line 13 my application blocks til process termination of test.py and
afterwards I get all outputs of test.py and miss the intermediate steps...

Here my test program:
test.py:
1 #!/usr/bin/python
2 import time
3 cnt = 5
4 while cnt:
5 time.sleep(1)
6 print "progress"
7 cnt-=1

Every 1 second I should get "progress" into my application but it does
not seem do work.
I played around with streaming operations and Threads but without success.

Any help are welcome!

Try with:

import time
import sys
cnt = 5
while cnt:
time.sleep(1)
print "progress"
sys.stdout.flush()
cnt-=1

Arne

 

[EJP]

I found out when the output of the external application exceed more than
64k then I get data from the stream. Hmmmm... It looks like here is a
"hidden" 64k buffer.

Precisely. In Python. Is there an API to flush the output?

 

[Harald Krammer]

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

Here is my code:
1 public class Main {
2 public static void main(String[] args) {
3 ProcessBuilder pb = new ProcessBuilder("/tmp/test.py");
4
5 try {
6 Process p = pb.start();
7 p = pb.start();
8
9 BufferedReader output = new BufferedReader(
10 new InputStreamReader(p.getInputStream()));
11 String line;
12
13 while ((line = output.readLine()) != null) {
14 System.out.println(line);
15 }
16
17 p.waitFor();
18 }
19 catch(Exception ex) {
20 ex.printStackTrace();
21 }
22 }
23}

In line 13 my application blocks til process termination of test.py and
afterwards I get all outputs of test.py and miss the intermediate
steps...

Here my test program:
test.py:
1 #!/usr/bin/python
2 import time
3 cnt = 5
4 while cnt:
5 time.sleep(1)
6 print "progress"
7 cnt-=1

Every 1 second I should get "progress" into my application but it does
not seem do work.
I played around with streaming operations and Threads but without
success.

Any help are welcome!

Try with:

import time
import sys
cnt = 5
while cnt:
time.sleep(1)
print "progress"
sys.stdout.flush()
cnt-=1

Arne

Hello,
yes it helps, but in Emacs it works without any flushes. That means I
can parse the same python program in "real time" without
'sys.stdout.flush()'.
I will investigate the differences between Emacs Lisp and Java (system
calls).

Harald

--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[cbossens73]

Hello *,

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

I don't know it this could be of any help but I've been working
a *lot* with executing external processes from Java, on several
OSes (Linux, OS X and lots of Windows flavors) and this way before
ProcessBuilder came to exist.

I had total control on the shell/batch scripts so what I did
was to make sure I'd never have a script produce anything on
stdout nor on stderr nor need anything on stdin.

And you can also always wrap a script you have no control on
into another script that takes care of redirecting the output.

On Linux / OS X I'd use nohup (man nohup - run a command
immune to hangups, with output to a non-tty), out of memory
something like this:

nohup $@ > $TEMP_REDIR_FILE 2>&1 < /dev/null

(AFAIK you cannot invoke such a command line directly
from Java, this has to be wrapped in a script)

From Java I'd then not bother at all with streams/buffer,
nor consuming the stdout and stderr etc. I'd simply parse
then delete the temporary output file.

Once again I don't know if it helps or not in your case:
in my case the shell script were producing well defined
output and the output file was not growing over time.

But for me it worked, performance wasn't an issue, and
it was way simpler to use than messing with correctly
parsing the streams on different OSes etc.

This group is full of post made by people over the years
that have had problem when parsing stdout/stderr when
launching external processes from Java.

I remember that many things were breaking apart and breaking
apart differently depending on the OS versions (I particularly
remember joy "debugging" Windows 2000 weirdities, but that was
a long time ago).

Once we switched to the radical "no stdout/no stderr" output
at all things proved to be much easier.

 

[Roedy Green]

I would like to parse in "real time" an output of a external program in
my Java application, but I only get the output of my program after
program termination.

you must use several threads. See
http://mindprod.com/jgloss/exec.html

However, I got his email today, which suggests it may be more
complicated than that:

"I found your example of using the Java ProcessBuilder very helpfully.
However; the example shows starting a new thread to consume input from
the Process that ProcessBuilder started. I've found that running
having a separate thread to tend to the "child" can lead to an Stream
Closed Exception being thrown by the line "while ( ( line =
br.readLine() ) != null )". My guess is that in the JVM this
comsumer thread is getting ahead of the the child and the input stream
isn't opened completely.

I've found that on slow CPU's having the extra thread works ok. But
on faster CPU I've had problems with the BufferedReader. I'm
calling Processbuilder start from a ScheduledExecutorService instance
at a one a second. The BufferedReader consumer thread fails after a
variable number of calls from the ScheduledExecutorService. Something
unfriendly is happening in JVM."
--
Roedy Green Canadian Mind Products
http://mindprod.com

"At this point, 29 percent of fish and seafood species have collapsed - that is,
their catch has declined by 90 percent. It is a very clear trend, and it is accelerating.
If the long-term trend continues, all fish and seafood species are projected to collapse
within my lifetime -- by 2048."
~ Dr. Boris Worm of Dalhousie University

 

[Arne Vajhøj]

I thought the problem was Python not flushing output when connected to a
pipe rather than to a console.

It was.

But Roedy seems to have skipped those posts.

Arne

 

[Arne Vajhøj]

I found my Fedora Core 5 system duplicates this problem. Python has a
parameter "-u" for running unbuffered. Change your script's first line to

#!/usr/bin/python -u

It works fine on Windows as well !

This must be *the* solution.

Arne

 

[Arne Vajhøj]

From Java I'd then not bother at all with streams/buffer,
nor consuming the stdout and stderr etc. I'd simply parse
then delete the temporary output file.

Once again I don't know if it helps or not in your case:
in my case the shell script were producing well defined
output and the output file was not growing over time.

That is a solution.

But I don't think it will provide the real time effect
that the original poster was asking for.

He was getting everything he wanted, but it just all came
at once at the end instead of by regular intervals.

I can not see a temporary file solve that.

Arne

 

[Harald Krammer]

I found my Fedora Core 5 system duplicates this problem. Python has a
parameter "-u" for running unbuffered. Change your script's first line to

#!/usr/bin/python -u

It works with '-u' and will help to prevent any code modifications in
the scripts.

--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[Harald Krammer]

It works fine on Windows as well !

This must be *the* solution.

I am not sure...
Why '-u' isn't necessary in Emacs for live stream analyses?

The Problem will happen also with following c program:
#include <stdio.h>
#include <unistd.h>

int main() {
int cnt;

for (cnt= 0; cnt<5; cnt++) {
printf("progress\n");
sleep(1);
}

return 0;
}


I compared the system calls between Emacs and Java, but I saw to many
differences, so it's not comparable......
I will try the examples from Roedy (http://mindprod.com/jgloss/exec.html)


Harald

--

Harald Krammer
Brucknerstrasse 33
A - 4020 Linz
AUSTRIA

Mobil +43.(0) 664. 130 59 58
Mail: Harald.Krammer (at) hkr.at

 

[Arne Vajhøj]

I am not sure...
Why '-u' isn't necessary in Emacs for live stream analyses?

I compared the system calls between Emacs and Java, but I saw to many
differences, so it's not comparable......

Given that the need to explicit do something to flush output
is a common feature in many languages, then it seems a waste of
time to focus on how Emacs apparently does something very special
when it runs a Python program.

Arne