Java Runtime.exec problem

Discussion in 'Java' started by suru, Jul 16, 2009.

  1. suru

    suru Guest

    HI,
    I am using Runtime.exec to excute my unix (wrapper) commands.
    These commands are fired consecutively.
    But somtimes i get following exception,
    java.io.IOException: Stream closed
    at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:
    145)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:304)
    at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:
    411)
    at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:
    453)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    at java.io.InputStreamReader.read(InputStreamReader.java:167)
    at java.io.BufferedReader.fill(BufferedReader.java:136)
    at java.io.BufferedReader.readLine(BufferedReader.java:299)
    at java.io.BufferedReader.readLine(BufferedReader.java:362)
    at com.vistaar.web.util.StreamGobbler.run(ToolkitAgent.java:711)

    Following is the code snippet:
    //method from which exec is invoked
    public static String processCommandWithNewLine(String
    commandToExecute) {
    Runtime systemShell = Runtime.getRuntime();
    Process outputProcess = null;
    String toolkitOutput = "";
    try {
    LogHelper.debug(lg, commandToExecute);
    outputProcess = systemShell.exec(commandToExecute);

    StreamGobbler errorGobbler = new StreamGobbler
    (outputProcess.getErrorStream(), true, lg);
    StreamGobbler outputGobbler = new StreamGobbler
    (outputProcess.getInputStream(), true, lg);
    errorGobbler.start();
    outputGobbler.start();
    outputProcess.waitFor();

    toolkitOutput = toolkitOutput + outputGobbler.getOutput();
    toolkitOutput = toolkitOutput + errorGobbler.getOutput();
    LogHelper.debug(lg, toolkitOutput);
    } catch (Throwable ex) {
    LogHelper.printErrorTrace(lg, ex);
    } finally {
    closeProcessStreams(outputProcess);
    }
    return toolkitOutput.trim();
    }

    // steam gobbler implmentaion to read output of the command
    class StreamGobbler extends Thread {

    InputStream is;
    String output = "";
    boolean insertNewLine;
    Logger lg;

    StreamGobbler(InputStream is, boolean insertNewLine, Logger lg) {
    this.is = is;
    this.insertNewLine = insertNewLine;
    this.lg = lg;
    }

    public void run() {
    try {
    InputStreamReader isr = new InputStreamReader(is);
    BufferedReader br = new BufferedReader(isr);
    String line = null;
    if (insertNewLine) {
    while ((line = br.readLine()) != null) {
    output = output + line + "\n";
    }
    } else {
    while ((line = br.readLine()) != null) {
    output = output + line;
    }
    }
    } catch (IOException ioe) {
    LogHelper.printErrorTrace(lg, ioe);
    }
    }

    public String getOutput() {
    return output;
    }
    }


    1. i m using thread to read output & error stream simultaneously
    2. the above problem doesn't come consistently, but comes frequently

    eventhough the program says "stream closed",
    when i fire same command from shell it gives me proper output

    any idea what is causing the problem?
    or der there any better way to read output given by exec process?

    thnx,
    suraj
    suru, Jul 16, 2009
    #1
    1. Advertising

  2. On Jul 16, 5:19 pm, suru <> wrote:
    >..
    > Following is the code snippet:


    For better chance of getting help, post an SSCCE.
    <http://pscode.org/sscce.html>

    --
    Andrew T.
    pscode.org
    Andrew Thompson, Jul 16, 2009
    #2
    1. Advertising

  3. suru

    suru Guest

    On Jul 16, 3:48 pm, Steven Simpson <> wrote:
    > suru wrote:
    > > I am using Runtime.exec to excute my unix (wrapper) commands.
    > > These commands are fired consecutively.
    > > But somtimes i get following exception,
    > > java.io.IOException: Stream closed

    >
    > Okay something closed your stream, but your snippet doesn't show anything
    > explicitly doing that.  However...
    >
    >
    >
    >
    >
    > >             outputProcess = systemShell.exec(commandToExecute);

    >
    > >             StreamGobbler errorGobbler = new StreamGobbler
    > > (outputProcess.getErrorStream(), true, lg);
    > >             StreamGobbler outputGobbler = new StreamGobbler
    > > (outputProcess.getInputStream(), true, lg);
    > >             errorGobbler.start();
    > >             outputGobbler.start();
    > >             outputProcess.waitFor();

    >
    > >             toolkitOutput = toolkitOutput + outputGobbler..getOutput();
    > >             toolkitOutput = toolkitOutput + errorGobbler.getOutput();
    > >             LogHelper.debug(lg, toolkitOutput);
    > >         } catch (Throwable ex) {
    > >             LogHelper.printErrorTrace(lg, ex);
    > >         } finally {
    > >             closeProcessStreams(outputProcess);

    >
    > Does this last line close the streams that your gobblers are gobbling
    > on?  If so, they might still be gobbling after the process has finished
    > (i.e. after waitFor() returns).  Before closing the streams, try waiting
    > for the gobblers to complete (Thread#join()?), or get the gobblers to
    > close their respective streams themselves.
    >
    > --
    > ss at comp dot lancs dot ac dot uk- Hide quoted text -
    >
    > - Show quoted text -



    really good opinion :)
    wil try and let u know
    suru, Jul 16, 2009
    #3
  4. suru

    suru Guest

    i tried with Thread.join approach,
    for some calls it just hangs after join call :(
    suru, Jul 16, 2009
    #4
  5. suru

    Lew Guest

    suru wrote:
    > i [sic] tried with Thread.join approach,
    > for some calls it just hangs after join call :(


    Did the thread it was joining terminate? It sounds like it didn't.
    The join() won't return until the thread it's joining terminates.

    Also, join() is a blunt tool.

    It's really not possible to fully answer your question without seeing
    the code, though.

    http://sscce.org/

    --
    Lew
    Lew, Jul 16, 2009
    #5
  6. suru

    Lew Guest

    On Jul 16, 10:29 am, Steven Simpson <> wrote:
    > suru wrote:
    > > i tried with Thread.join approach,
    > > for some calls it just hangs after join call :(

    >
    > Where did you put the join()s?  Which one is hanging?  Can you determine
    > whether the gobblers have completed?  What does your code look like
    > now?  (Take note of others' comments about SSCCEs too.)
    >
    > From your original post:
    >
    > >             outputProcess.waitFor();

    >
    > >             toolkitOutput = toolkitOutput + outputGobbler..getOutput();
    > >             toolkitOutput = toolkitOutput + errorGobbler.getOutput();

    >
    > I think you're going to have to do some sort of waiting (e.g. join())
    > between waitFor() and getOutput() anyway, as you need to make sure that
    > your gobblers have finished by then.  You could make getOutput() block
    > until the stream is finished, instead of using join().
    >
    > Get the gobblers (instead of anything else) to close the streams too.
    > There's no reason for anything else to be dealing with the streams (is
    > there?).
    >


    Also you might consider using an Executor and/or a Future
    (FutureTask).

    Thread coordination is a fairly advanced topic. After decades of
    working with concurrent programming, I still keep researching,
    studying and repeating that research and study, and I still feel that
    I need to keep doing so.

    I recommend the book /Java Concurrency in Practice/ by Brian Goetz, et
    al. There are articles by Mr. Goetz available on the IBM
    DeveloperWorks site that will help a lot. The Sun tutorials are a
    decent starting point.

    --
    Lew
    Lew, Jul 16, 2009
    #6
  7. suru

    Guest

    Hi Suraj,

    On Jul 16, 8:19 am, suru <> wrote:
    ....
    > any idea what is causing the problem?


    I've long stopped caring about such problems. Java's
    runtime support as always been flakey and tricky (documented
    bugs regarding sometimes magically disappearing slave processes
    when using the Un*x pipe '|' symbol comes to mind).

    Correctly running external processes was already a "Java Pitfall"
    ten years ago, and it still is. Your post is a case in point.

    The sheer number of posts in here with people having
    trouble executing external processes is unusually high.

    The number of framework and "helper" classes out there trying
    to give a saner abstraction is also madenning (and directly
    related to the immediate pain most people encounter when trying
    to launch external processes from Java).

    I've probably launched more external processes, on more
    platforms/OSes/differents versions of these OSes, than
    anyone in this group and I can give you the "trick" I'm
    using since years.


    > or der there any better way to read output given by exec process?


    In my opinion yes, and it's trivially simple...

    "free your mind"

    Seen that correctly "Gobbling" or consuming stdout/stderr
    from Java is tricky, I went on with a radical, ultra-simple,
    solution : I wrap all my scripts in another wrapper script
    that suppresses every stdout/stderr output and redirects them
    to temporary files. In other words, I let the OS take care
    of the correct gobbling/consuming of the script's output
    by redirecting these to temporary files.

    Then I simply parse these temporary files.

    This is both radical and elegant and it works on Windows, OS X,
    Linux, and other Un*x flavors.

    The Un*x shell script (Bash in this case but YMMV) may
    look like this (out of memory, to give an idea [both stdout
    and stderr are redirected here to the same temp file]):

    #!/bin/bash
    TEMP_REDIR_FILE=$1
    shift;
    $@ > $TEMP_REDIR_FILE 2>&1

    Brutally simple.

    The equivalent for the various flavor of Windows is
    easy too.

    In case you need to specifically output to a non-tty
    and be immune to hangups, then 'nohup' is useful
    (man nohup and check your shell's nohup doc).

    My actual scripts are a tad bit more advanced for in
    the case of terminating external processes (most do
    terminate, some are 'non-stop') I also append some
    "magic" string to know, when parsing my file, that
    the command is done, and exited without errors, etc.

    I abstracted all this in a clean interface and handle
    the result (ie the parsed stdout/stderr text) using a
    callback. I implemented timeout facilities, watchdog
    process kill -9'ing the watched process should it block
    (a correctly implement Un*x OS makes guarantees regarding
    kill -9's behavior [on Windows this was more complicated,
    I had to resort to pskill.exe]), I also implemented retries,
    etc.

    It's just an opinion but I find this more advanced than what
    you can do in Java (for example regarding the "kill -9"
    guarantees [longstanding Java bugs where Process's destroy()
    would not really release all the ressources], yadda yadda!) but
    yet I find my abstraction much more simple to
    use than the 'external processes Java SNAFU' (pardon, I meant
    than the Java API dealing with external processes :)

    I wrote this several years ago and it never failed on me,

    Alex


    --
    "You see things; and you say, 'Why?
    "But I dream things that never were; and I say, 'Why not?'"
    , Jul 17, 2009
    #7
  8. suru

    suru Guest

    really nice opinion Alex,

    i hv few queries for understanding,

    1. do u know why exactly java hangs up? is der any posts/link
    explaining the details?
    Also, i have observed that sometimes the program hangs at
    "br.readLine()" call

    2. in ur suggestion, u have mentioned that, u redirect ur process
    output to a temporary file,
    is der simple way to keep temporary file name unique? since multiple
    users may execute same commands (from java) hence output for each user
    should be in different files.

    thnx,
    suraj
    suru, Jul 21, 2009
    #8
  9. On Jul 21, 4:40 pm, suru <> wrote:
    > really nice opinion Alex,
    >
    > i hv .. u ... der .. ur ...


    Do you expect anyone to read that rubbish?

    How about you show your appreciation for the
    help you've received by showing the extra effort
    needed to type words properly (and in the process,
    you might save yourself from seeming like a lazy
    bozo).

    --
    Andrew T.
    Andrew Thompson, Jul 21, 2009
    #9
  10. suru

    suru Guest

    apologies for using short words (got habit due to chat) :)


    Following is the proper version,

    Really nice opinion Alex,

    I have few queries for understanding,


    1. Do you know why exactly java hangs up? is there any posts/link
    explaining the details?
    Also, i have observed that sometimes the program hangs at
    "br.readLine()" call


    2. In your suggestion, you have mentioned that, you redirect your
    process
    output to a temporary file,
    is there simple way to keep temporary file name unique? since
    multiple
    users may execute same commands (from java) hence output for each
    user
    should be in different files.


    Thanks,
    Suraj
    suru, Jul 21, 2009
    #10
  11. suru

    Roedy Green Guest

    On Thu, 23 Jul 2009 14:15:51 +0100, Steven Simpson <>
    wrote, quoted or indirectly quoted someone who said :

    >> 2. In your suggestion, you have mentioned that, you redirect your process
    >> output to a temporary file, is there simple way to keep temporary file name unique?

    >
    ><http://java.sun.com/javase/6/docs/api/java/io/File.html#createTempFile%28java.lang.String,%20java.lang.String%29>


    If you want your temp file created in a particular directory so it can
    be easily renamed to the original file, use the createTempFile as part
    of the hunkio package. See http://mindprod.com/products1.htmlHUNKIO
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    "The industrial civilisation is based on the consumption of energy resources that are inherently limited in quantity, and that are about to become scarce. When they do, competition for what remains will trigger dramatic economic and geopolitical events; in the end, it may be impossible for even a single nation to sustain industrialism as we have know it in the twentieth century."
    ~ Richard Heinberg, The Party’s Over: Oil, War, and the Fate of Industrial Societies
    Roedy Green, Jul 23, 2009
    #11
  12. suru

    Arne Vajhøj Guest

    Roedy Green wrote:
    > On Thu, 23 Jul 2009 14:15:51 +0100, Steven Simpson <>
    > wrote, quoted or indirectly quoted someone who said :
    >
    >>> 2. In your suggestion, you have mentioned that, you redirect your process
    >>> output to a temporary file, is there simple way to keep temporary file name unique?

    >> <http://java.sun.com/javase/6/docs/api/java/io/File.html#createTempFile%28java.lang.String,%20java.lang.String%29>

    >
    > If you want your temp file created in a particular directory so it can
    > be easily renamed to the original file, use the createTempFile as part
    > of the hunkio package. See http://mindprod.com/products1.htmlHUNKIO


    Why ?

    Given that:

    http://java.sun.com/javase/6/docs/a....lang.String, java.lang.String, java.io.File)
    has existed since 1.2 !

    Arne
    Arne Vajhøj, Jul 24, 2009
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Charlie Zhang
    Replies:
    5
    Views:
    6,518
    Québec
    Jun 19, 2004
  2. seenuvas
    Replies:
    7
    Views:
    2,645
    Roedy Green
    Jan 24, 2006
  3. seenuvas
    Replies:
    0
    Views:
    513
    seenuvas
    Jan 24, 2006
  4. Hal Vaughan
    Replies:
    11
    Views:
    1,095
    Gordon Beaton
    May 22, 2006
  5. Random Task
    Replies:
    12
    Views:
    633
    Joe Smith
    Dec 4, 2005
Loading...

Share This Page