Re: Capturing stdout from shell script

Discussion in 'Java' started by Nigel Wade, Mar 15, 2011.

  1. Nigel Wade

    Nigel Wade Guest

    On 15/03/11 13:46, Mark wrote:
    > Hi,
    >
    > We have the following code to call a shell script:
    >
    > Runtime rt = Runtime.getRuntime();
    > Process proc = rt.exec(scriptName );
    > int exitVal = proc.waitFor();
    > log.info("Returned Value is: " + exitVal);
    >
    > How can get access to the output from the shell script (normally from
    > stdout)?


    You may be better off these days using ProcessBuilder, and combining
    stdout and stderr using ProcessBuilder.redirectErrorStream(). Then all
    you need do is use the Process.getInputStream() to read both stdout and
    stderr.

    If you stick to using Runtime.exec() you'll need to use threads to read
    both the stdout and stderr simultaneously to avoid potential deadlock
    due to the buffered nature of the streams.

    --
    Nigel Wade
     
    Nigel Wade, Mar 15, 2011
    #1
    1. Advertising

  2. In message <>, Nigel Wade wrote:

    > If you stick to using Runtime.exec() you'll need to use threads to read
    > both the stdout and stderr simultaneously to avoid potential deadlock
    > due to the buffered nature of the streams.


    Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?
     
    Lawrence D'Oliveiro, Mar 15, 2011
    #2
    1. Advertising

  3. Lawrence D'Oliveiro <_zealand> wrote:
    > In message <>, Nigel Wade wrote:
    >> If you stick to using Runtime.exec() you'll need to use threads to read
    >> both the stdout and stderr simultaneously to avoid potential deadlock
    >> due to the buffered nature of the streams.

    > Don’t you have select(2) <http://docs.python.org/py3k/library/select.html>?


    Iirc, with nio Java offers also the non-blocking approach to I/O.
    Haven't yet used it myself, yet, as I do such tasks in Tcl ;-)

    Based on how fragile thread-programming can be (forget one "synchronized"
    and you end up reading a local CPU's cache instead of shared memory. Put
    bad ones in and face bad performance or deadlocks), I understand this
    wish for select-based IO - yet it *is* already available in Java.

    Read up on java.nio.channels.Selector for a start.

    PS: to be fair: event-based IO also has its pitfalls and hardships,
    just different ones. ;)
     
    Andreas Leitgeb, Mar 16, 2011
    #3
  4. In message <>, Andreas Leitgeb
    wrote:

    > Read up on java.nio.channels.Selector for a start.


    <http://developer.android.com/reference/java/nio/channels/Selector.html>

    Bloody hell. How many different layers of classes and calls do you need in
    Java to implement such a simple concept?

    The underlying POSIX service is so simple
    <http://linux.die.net/man/2/select>: pass 3 bitmasks of file descriptors to
    wait for, plus an optional timeout. Python keeps this simplicity
    <http://docs.python.org/py3k/library/select.html>: pass 3 sequences of
    objects which can be anything you like, so long as they have valid “filenoâ€
    methods.
     
    Lawrence D'Oliveiro, Mar 17, 2011
    #4
  5. Lawrence D'Oliveiro <_zealand> wrote:
    > In message <>, Andreas Leitgeb wrote:
    >> Read up on java.nio.channels.Selector for a start.

    > <http://developer.android.com/reference/java/nio/channels/Selector.html>
    > Bloody hell. How many different layers of classes and calls do you need in
    > Java to implement such a simple concept?


    "simple". Yes, that describes it. Too simple in some cases, where lots of
    filedescriptors are totally used by the app. (select scales badly with the
    value of the latest filedescriptor, even if only a small number of fds is
    really selected on.) There's also "poll" which is not that "simple", but
    doesn't care if the fds you wait for are 3,4,5 or 1000,4042,4043.

    I have no idea, which of select or poll is used with java's nio Selector,
    but I believe that it is not user's concern, anyway. Just because the user
    doesn't get to build up bitsets, himself, its probably a good abstraction
    that would allow even for a dynamic (or at least system-dependent) choice
    between select and poll.

    Tcl also uses select, internally, but the script-interface hides this away
    so it could probably be rewritten for poll any time. Also, it allows for
    easier porting to systems, where such notifications for channels are handled
    entirely different.
     
    Andreas Leitgeb, Mar 17, 2011
    #5
  6. In message <>, Andreas Leitgeb
    wrote:

    > Lawrence D'Oliveiro <_zealand> wrote:
    >
    >> <http://developer.android.com/reference/java/nio/channels/Selector.html>
    >> Bloody hell. How many different layers of classes and calls do you need
    >> in Java to implement such a simple concept?

    >
    > "simple". Yes, that describes it. Too simple in some cases, where lots of
    > filedescriptors are totally used by the app. (select scales badly with the
    > value of the latest filedescriptor, even if only a small number of fds is
    > really selected on.)


    Really? I’ve written servers in Perl, C++ and Python using select, and its
    performance was never a problem; any bottleneck turned out to be elsewhere.

    > I have no idea, which of select or poll is used with java's nio Selector,
    > but I believe that it is not user's concern, anyway. Just because the
    > user doesn't get to build up bitsets, himself, its probably a good
    > abstraction that would allow even for a dynamic (or at least
    > system-dependent) choice between select and poll.


    Python manages to avoid the bitsets in a much simpler way.

    > Tcl also uses select, internally ...


    Tcl is an antiquated language, where arrays are not even first-class
    objects.
     
    Lawrence D'Oliveiro, Mar 18, 2011
    #6
  7. Lawrence D'Oliveiro <_zealand> wrote:
    >> [Andreas Leitgeb wrote about limitations of select]

    > Really? I’ve written servers in Perl, C++ and Python using select, and its
    > performance was never a problem; any bottleneck turned out to be elsewhere.

    What were the access-rates on those servers? How many fds did it ever have
    open at the same time? (including fds that were not incoming connections)

    > Python manages to avoid the bitsets in a much simpler way.

    That sounds good. Maybe it is closer to tcl, than I thought it was
    after reading your mention of explicit "select" in python?

    >> Tcl also uses select, internally ...

    > Tcl is an antiquated language, where arrays are not even first-class
    > objects.

    Apart from that it is a bogus statement, it also doesn't address
    Tcl's approach to event-based programming, namely hiding that ugly
    select system-call beyond recognition, which was the only reason
    I brought it up here.
     
    Andreas Leitgeb, Mar 18, 2011
    #7
  8. Lawrence D'Oliveiro wrote:
    > In message <>, Andreas Leitgeb
    > wrote:
    >
    > Really? I’ve written servers in Perl, C++ and Python using select, and its
    > performance was never a problem; any bottleneck turned out to be elsewhere.


    The scalability problems with select are well-documented. See for
    example Stevens, _UNIX Network Programming_ 2nd ed vol 1, 6.3; that
    also refers you to Wright & Stevens, _TCP/IP Illustrated_ vol 2,
    16.13, which shows an implementation of select that may be enlightening.

    UNP 6.3 also notes that in some implementations attempting to enlarge
    the select FD_SETSIZE does not work correctly because the sets were
    still restricted in the kernel; the Standard Unix Specification (v3)
    does not require that implementations permit changing FD_SETISZE.

    UNP 6.10 notes that poll() does not suffer from the latter problem.

    Another discussion may be found at [1], which notes that neither
    select() nor poll() scale very well to thousands of descriptors;
    that's why OS vendors started introducing other mechanisms, such as
    /dev/poll, in the mid-1990s.

    Another example is the "Heuristic 2" discussion in the Winsock FAQ's
    discussion of I/O strategies.[2] Of course, select in Winsock uses an
    array of handles rather than a bitmask, so it's closer to Unix's
    poll() than to the traditional select() in its implementation.

    Or see the discussion at [3], particularly where Barry Margolin points
    out that there are different scaling tradeoffs between select() and
    poll(). If the descriptor set is large and dense, select() is more
    efficient in switching between user and kernel mode; if either of
    those doesn't hold, poll() is more efficient at that point.

    The scalability issues with select() ought to be obvious. There's the
    limit of FD_SETSIZE, in implementations where that limit is in the
    kernel as well. (If it isn't, you can work around that at the
    application level.) There's the cost of copying FD_SETs between user
    and kernel space. There's the cost of linear scans of the FD_SETs. The
    latter two are particularly wasteful if the sets are sparse; that's
    the problem that poll() fixes, but at the cost of using far more
    storage per descriptor, so whether you benefit from poll() depends on
    just which descriptors you're using.

    On Windows, there's just select(), and it's effectively poll(), and
    (as the Winsock FAQ explains) it's pretty much the worst of the
    available choices.

    For a handful of clients, even up to several hundred, it likely
    doesn't matter; chances are good that I/O multiplexing won't dominate
    processing time, even though it adds a lot of looping within the main
    loop. When a server has to handle thousands of simultaneous clients
    it's a different story. I have (proprietary) profiling data from real,
    commercial servers showing significant performance issues with
    select() at those kinds of loads.


    [1] http://www.kegel.com/c10k.html
    [2] http://tangentsoft.net/wskfaq/articles/io-strategies.html
    [3]
    https://groups.google.com/group/com...read/thread/24409068a4c51c62/e3cdcda1e6c216f1

    --
    Michael Wojcik
    Micro Focus
    Rhetoric & Writing, Michigan State University
     
    Michael Wojcik, Mar 18, 2011
    #8
  9. In message <>, Michael Wojcik wrote:

    > UNP 6.10 notes that poll() does not suffer from the latter problem.


    Fine. I’ve never used poll, but Python supports that too if you want, with
    just one class and four methods
    <http://docs.python.org/py3k/library/select.html#poll-objects>.

    Contrast that to the number of different classes that Java requires
    <http://developer.android.com/reference/java/nio/channels/Selector.html>:
    Selector (9 methods), SelectableChannel (8 methods), SelectionKey (another
    dozen methods), SelectorProvider ... I’m losing count.
     
    Lawrence D'Oliveiro, Mar 18, 2011
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeff Learman

    shell script stdout to python string?

    Jeff Learman, May 20, 2004, in forum: Python
    Replies:
    2
    Views:
    1,405
    Jeff Learman
    May 20, 2004
  2. Christian Heimes
    Replies:
    0
    Views:
    630
    Christian Heimes
    Feb 27, 2008
  3. Gerardo Herzig
    Replies:
    1
    Views:
    1,141
    Philipp Pagel
    Feb 27, 2008
  4. D'Arcy J.M. Cain
    Replies:
    0
    Views:
    893
    D'Arcy J.M. Cain
    Feb 27, 2008
  5. moongeegee

    execute a shell script in a shell script

    moongeegee, Dec 3, 2007, in forum: Perl Misc
    Replies:
    2
    Views:
    276
    Ben Morrow
    Dec 4, 2007
Loading...

Share This Page