How can you make idle processors pick up java work?

Discussion in 'Java' started by qwertmonkey@syberiaoutpost.ru, Jul 31, 2012.

  1. Guest

    >> Is there a way to make these processors pick up/share work also, or
    >> do you have to use some sort of scheduling framework on top of java?


    > Use multiple threads?

    ~
    a) I need to actually scan large text files (10+ million lines).
    b) On each line there is a NL sentence.
    c) That processing should be run only once, but as fast as possible.
    ~
    d) If you go:
    d.1) int iPrx = Runtime.getRuntime().availableProcessors();
    d.2) count all lines
    d.3) split the file in (total lines)/iPrx
    d.4) then run iPrx threads (or executable instances using a batch script)
    the time you waste on d.2) and d.3) will make all that strat senseless
    ~
    I have no way to influence how those large files are generated
    ~
    e) because of the large sizes of the files you can't even go
    ~
    FIS = new FileInputStream(IFl);
    FileChannel IFlChnl = FIS.getChannel();
    int iChnlSz = (int)IFlChnl.size();
    MappedByteBuffer MptBytBfr = IFlChnl.map(FileChannel.MapMode.READ_ONLY, 0, iChnlSz);
    ~
    so, apparently, the only option I have is:
    ~
    BfR = Files.newBufferedReader(DirPth, ChrStUTF8);
    String aSx = BfR.readLine();
    while(aSx != null){

    aSx = BfR.readLine();
    }
    ~
    do you know of a faster way to go about this?
    ~
    lbrtchx
     
    , Jul 31, 2012
    #1
    1. Advertising

  2. David Lamb Guest

    On 30/07/2012 7:27 PM, wrote:
    >>> Is there a way to make these processors pick up/share work also, or
    >>> do you have to use some sort of scheduling framework on top of java?

    >
    >> Use multiple threads?

    > ~
    > a) I need to actually scan large text files (10+ million lines).
    > b) On each line there is a NL sentence.
    > c) That processing should be run only once, but as fast as possible.
    > ~
    > d) If you go:
    > d.1) int iPrx = Runtime.getRuntime().availableProcessors();
    > d.2) count all lines
    > d.3) split the file in (total lines)/iPrx
    > d.4) then run iPrx threads (or executable instances using a batch script)
    > the time you waste on d.2) and d.3) will make all that strat senseless


    How slow is the NL processing? Does it make any sense to read lines in
    one thread and pass each off to one of the iPrx-1 other threads that
    might run on separate processors?
     
    David Lamb, Jul 31, 2012
    #2
    1. Advertising

  3. [Gah, your newsreader is incapable of threading posts correctly. Please
    find a non-broken one.]

    On 7/30/2012 7:27 PM, wrote:
    >>> Is there a way to make these processors pick up/share work also, or
    >>> do you have to use some sort of scheduling framework on top of java?

    >
    >> Use multiple threads?

    > ~
    > a) I need to actually scan large text files (10+ million lines).
    > b) On each line there is a NL sentence.
    > c) That processing should be run only once, but as fast as possible.


    Only 10M-line files?

    The easiest way to do this is to just make a ThreadPoolExecutor and have
    your main thread dispatch requests as fast as possible to the pool. Or
    you can do the work pooling yourself, which may be faster since you're
    not continually posting Runnable's, but timing results would be
    necessary to convince me.

    There are other options, but chances are, your disk drive is going to
    saturate first (in short, it involves reading non-consecutive pages of
    the file, which is generally a recipe for disaster).

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
     
    Joshua Cranmer, Jul 31, 2012
    #3
  4. In article <jv759b$s7$>,
    wrote:

    > d.2) count all lines


    Maybe ask ProcessBuilder to `wc -l`, or similar?

    --
    John B. Matthews
    trashgod at gmail dot com
    <http://sites.google.com/site/drjohnbmatthews>
     
    John B. Matthews, Jul 31, 2012
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    237
    Roedy Green
    Jul 31, 2012
  2. Replies:
    2
    Views:
    251
    Joshua Cranmer
    Jul 31, 2012
  3. Replies:
    0
    Views:
    202
  4. Replies:
    0
    Views:
    139
  5. lbrtchx
    Replies:
    3
    Views:
    190
    Roedy Green
    Jul 31, 2012
Loading...

Share This Page