Runtime.getRuntime().exec() very slow in Java program.

Discussion in 'Java' started by au.danji@gmail.com, Apr 16, 2009.

  1. Guest

    I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    files on a linux system.
    but the getRuntime.exec() is very slow, it can only copy around 2-10
    documents/second to my target directory when I have 1000 files. Can
    anyone give some suggestions about my code below? thanks.

    for (doc ahit : docList) {

    try{
    shellCMD="cp "+ srcDir + "/"+ ahit.doc_id + " " +
    tarDumpDir; //copy xml to tmp folder
    Process process = Runtime.getRuntime().exec
    (shellCMD);
    //process.waitFor();
    process.getInputStream().close();
    process.getOutputStream().close();
    process.getErrorStream().close();
    }
    catch (Exception e){
    logError("Copy XML fail: " + e);
    }
    }
     
    , Apr 16, 2009
    #1
    1. Advertising

  2. Lew Guest

    wrote:
    > I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    > files on a linux system.
    > but the getRuntime.exec() is very slow, it can only copy around 2-10
    > documents/second to my target directory when I have 1000 files.


    How fast should it be?

    Why do you think it should be that fast instead?

    > Can anyone give some suggestions about my code below? thanks.
    >
    > for (doc ahit : docList) {
    >
    >             try{
    >                 shellCMD="cp "+ srcDir + "/"+ ahit.doc_id + " " +
    > tarDumpDir;  //copy xml to tmp folder
    >                 Process process = Runtime.getRuntime().exec
    > (shellCMD);
    >                 //process.waitFor();
    >                 process.getInputStream().close();
    >                 process.getOutputStream().close();
    >                 process.getErrorStream().close();
    >             }
    >             catch (Exception e){
    >                 logError("Copy XML fail: "  + e);
    >             }
    >         }


    I am only speculating, but several things occur to me.

    How fast would a shell script run:

    #!/bin/bash
    for fl in $*
    do
    bash cp ${srcDir}/${fl} ${tarDumpDir}/
    done
    ?

    Your program has to start a shell for each file copied. Given that
    you show us the "cp" command, presumably that shell has to process /
    etc/profile, ~/.profile and ~/.bashrc (or equivalent) each time, not
    to mention the scripts in /etc/profile.d/. Add to that the overhead
    of 'Runtime#exec()'.

    It would likely run faster if you either ran a single shell command to
    copy all the files, or used pure Java to do the copy without using
    'Runtime' at all.

    With a pure Java approach, you can put each copy in its own thread to
    achieve a measure of parallelism. Plus it would be portable.

    --
    Lew
     
    Lew, Apr 16, 2009
    #2
    1. Advertising

  3. wrote:
    > I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    > files on a linux system.
    > but the getRuntime.exec() is very slow, it can only copy around 2-10
    > documents/second to my target directory when I have 1000 files. Can
    > anyone give some suggestions about my code below? thanks.
    >
    > for (doc ahit : docList) {
    >
    > try{
    > shellCMD="cp "+ srcDir + "/"+ ahit.doc_id + " " +
    > tarDumpDir; //copy xml to tmp folder
    > Process process = Runtime.getRuntime().exec
    > (shellCMD);
    > //process.waitFor();
    > process.getInputStream().close();
    > process.getOutputStream().close();
    > process.getErrorStream().close();
    > }
    > catch (Exception e){
    > logError("Copy XML fail: " + e);
    > }
    > }


    Just use the Java buffered streams with large buffers to do your copying.

    --

    Knute Johnson
    email s/nospam/knute2009/

    --
    Posted via NewsDemon.com - Premium Uncensored Newsgroup Service
    ------->>>>>>http://www.NewsDemon.com<<<<<<------
    Unlimited Access, Anonymous Accounts, Uncensored Broadband Access
     
    Knute Johnson, Apr 16, 2009
    #3
  4. Lew Guest

    wrote:
    >> for (doc ahit : docList) {

    >


    BTW, 'doc' as a type name does not follow the Java naming conventions,
    which call for the first letter of type names to be upper case.

    <http://java.sun.com/docs/codeconv/index.html>

    A common and sensible variant of these conventions is to place an
    opening brace on its own line, indented to the same level as the
    control structure and the closing brace. Otherwise the Java community
    follows them rather closely. To do so promotes effective
    communication and helps minimize bugs.

    --
    Lew
     
    Lew, Apr 16, 2009
    #4
  5. Knute Johnson wrote on 16.04.2009 22:22:
    >
    > Just use the Java buffered streams with large buffers to do your copying.
    >

    Or use some ready-made class to do this, e.g. Apache Commons IO:

    <http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#copyDirectory(java.io.File,%20java.io.File,%20java.io.FileFilter)>
    or
    <http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#copyFile(java.io.File,%20java.io.File)>
     
    Thomas Kellerer, Apr 16, 2009
    #5
  6. Arne Vajhøj Guest

    wrote:
    > I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    > files on a linux system.
    > but the getRuntime.exec() is very slow, it can only copy around 2-10
    > documents/second to my target directory


    That does not say anything, because we do not know how big
    your files are.

    > when I have 1000 files. Can
    > anyone give some suggestions about my code below? thanks.
    >
    > for (doc ahit : docList) {
    >
    > try{
    > shellCMD="cp "+ srcDir + "/"+ ahit.doc_id + " " +
    > tarDumpDir; //copy xml to tmp folder
    > Process process = Runtime.getRuntime().exec
    > (shellCMD);
    > //process.waitFor();
    > process.getInputStream().close();
    > process.getOutputStream().close();
    > process.getErrorStream().close();
    > }
    > catch (Exception e){
    > logError("Copy XML fail: " + e);
    > }
    > }


    Both process creation and file creation are expensive operations. But
    they should be a lot faster than 2-10 per second.

    So if it is small files, then it should be faster.

    But there are a lot of parameters that influence it: available memory,
    other IO on the same disks etc..

    BTW, what is the purpose of getting all the streams and closing them?

    And not calling waitFor in the loop will help paralleization, but
    not waitFor at all makes the situation after the loop a bit
    fuzzy.

    Arne
     
    Arne Vajhøj, Apr 17, 2009
    #6
  7. Arne Vajhøj Guest

    Knute Johnson wrote:
    > wrote:
    >> I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    >> files on a linux system.


    > Just use the Java buffered streams with large buffers to do your copying.


    I would say unbuffered streams with a large user buffer.

    Arne
     
    Arne Vajhøj, Apr 17, 2009
    #7
  8. Jon Gómez Guest

    Lew wrote:
    > I am only speculating, but several things occur to me.
    >
    > How fast would a shell script run:
    >
    > #!/bin/bash
    > for fl in $*
    > do
    > bash cp ${srcDir}/${fl} ${tarDumpDir}/
    > done
    > ?


    You should remove "bash" in the line underneath "do", as it expects a
    script in that context and spawning another copy of bash is anyway
    un-necessary.

    Also, if the filenames have certain kinds of whitespace the shell
    variables should be quoted: "${fl}", etc. This can cause trouble,
    otherwise:

    touch tmp1
    hi="tmp1 tmp2"
    cp ${tmp1}
    # tmp2 now exists
    # expansion outside quotes
    # depends on IFS

    The OP could also straight-out shell glob in the call to exec(). Here's
    an example:

    Runtime.getRuntime().exec(
    new String[] {
    "sh", "-c", "cp ? to/"
    },
    null,
    null
    );

    You stole my thunder though, with the shell script suggestion :). I
    totally didn't even think of multithreading, I was so obsessed with the
    inefficiency of buffered I/O for a simple system call (SYS_rename?), but
    that's a great idea (especially if we want to be more portable).

    Jon.
     
    Jon Gómez, Apr 17, 2009
    #8
  9. Arne Vajhøj Guest

    Jon Gómez wrote:
    > Lew wrote:
    >> I am only speculating, but several things occur to me.
    >>
    >> How fast would a shell script run:
    >>
    >> #!/bin/bash
    >> for fl in $*
    >> do
    >> bash cp ${srcDir}/${fl} ${tarDumpDir}/
    >> done
    >> ?

    >
    > You should remove "bash" in the line underneath "do", as it expects a
    > script in that context and spawning another copy of bash is anyway
    > un-necessary.


    It is unnecessary functionality wise.

    But it is necessary to do the same as the Java program.

    Arne
     
    Arne Vajhøj, Apr 17, 2009
    #9
  10. Lew Guest

    Jon Gómez wrote:
    > You stole my thunder though, with the shell script suggestion :).


    The irony is that I am not suggesting a shell script.

    --
    Lew
     
    Lew, Apr 17, 2009
    #10
  11. Jon Gómez Guest

    Lew wrote:
    > Jon Gómez wrote:
    >> You stole my thunder though, with the shell script suggestion :).

    >
    > The irony is that I am not suggesting a shell script.


    True. I guess I read what I was thinking already in it.
    Jon.
     
    Jon Gómez, Apr 17, 2009
    #11
  12. Roedy Green Guest

    On Thu, 16 Apr 2009 13:04:36 -0700 (PDT), ""
    <> wrote, quoted or indirectly quoted someone who
    said :

    >I am trying to run the Runtime.getRuntime().exec(shellCMD) to copy
    >files on a linux system.
    >but the getRuntime.exec() is very slow, it can only copy around 2-10
    >documents/second to my target directory when I have 1000 files. Can
    >anyone give some suggestions about my code below? thanks.


    Use the FileTransfer class to copy files purely within Java. This will
    be much faster than spawning a command processor and copy. It will
    not have to load any new code for each copy.

    see http://mindprod.com/products1.html#FILETRANSFER

    If the files are fairly small, you can do it even faster with

    http://mindprod.com/products1.html#HUNKIO
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    Now for something completely different:
    http://www.youtube.com/watch?v=9lp0IWv8QZY
     
    Roedy Green, Apr 19, 2009
    #12
  13. Roedy Green Guest

    Roedy Green, Apr 19, 2009
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. MARTIN Herve \(EXT\)
    Replies:
    1
    Views:
    1,483
    Anthony Borla
    Nov 27, 2003
  2. Jakir
    Replies:
    4
    Views:
    3,233
    Jakir
    Sep 1, 2004
  3. knguyen
    Replies:
    9
    Views:
    2,852
    Roedy Green
    Feb 15, 2008
  4. moongeegee
    Replies:
    7
    Views:
    5,521
    Arne Vajhøj
    Oct 5, 2008
  5. SamuelXiao
    Replies:
    16
    Views:
    2,059
    Arne Vajhøj
    Sep 27, 2011
Loading...

Share This Page