How can you make idle processors pick up java work?

Discussion in 'Java' started by qwertmonkey, Jul 31, 2012.

  1. qwertmonkey

    qwertmonkey Guest

    From:

    >> Is there a way to make these processors pick up/share work also, or
    >> do you have to use some sort of scheduling framework on top of java?


    > Use multiple threads?

    ~
    a) I need to actually scan large text files (10+ million lines).
    b) On each line there is a NL sentence.
    c) That processing should be run only once, but as fast as possible.
    ~
    d) If you go:
    d.1) int iPrx = Runtime.getRuntime().availableProcessors();
    d.2) count all lines
    d.3) split the file in (total lines)/iPrx
    d.4) then run iPrx threads (or executable instances using a batch script)
    the time you waste on d.2) and d.3) will make all that strat senseless
    ~
    I have no way to influence how those large files are generated
    ~
    e) because of the large sizes of the files you can't even go
    ~
    FIS = new FileInputStream(IFl);
    FileChannel IFlChnl = FIS.getChannel();
    int iChnlSz = (int)IFlChnl.size();
    MappedByteBuffer MptBytBfr = IFlChnl.map(FileChannel.MapMode.READ_ONLY, 0,
    iChnlSz);
    ~
    so, apparently, the only option I have is:
    ~
    BfR = Files.newBufferedReader(DirPth, ChrStUTF8);
    String aSx = BfR.readLine();
    while(aSx != null){

    aSx = BfR.readLine();
    }
    ~
    do you know of a faster way to go about this?
    ~
    lbrtchx

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
     
    qwertmonkey, Jul 31, 2012
    #1
    1. Advertising

  2. To: qwertmonkey
    From: Patricia Shanahan <>

    On 7/30/2012 4:27 PM, wrote:
    >>> Is there a way to make these processors pick up/share work also, or
    >>> do you have to use some sort of scheduling framework on top of java?

    >
    >> Use multiple threads?

    > ~
    > a) I need to actually scan large text files (10+ million lines).
    > b) On each line there is a NL sentence.
    > c) That processing should be run only once, but as fast as possible.
    > ~
    > d) If you go:
    > d.1) int iPrx = Runtime.getRuntime().availableProcessors();
    > d.2) count all lines
    > d.3) split the file in (total lines)/iPrx
    > d.4) then run iPrx threads (or executable instances using a batch script)
    > the time you waste on d.2) and d.3) will make all that strat senseless


    Why worry about splitting by actual line count, rather than by byte position in
    file?

    Patricia

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
     
    Patricia Shanahan, Jul 31, 2012
    #2
    1. Advertising

  3. qwertmonkey

    David Lamb Guest

    To: qwertmonkey
    From: David Lamb <>

    On 30/07/2012 7:27 PM, wrote:
    >>> Is there a way to make these processors pick up/share work also, or
    >>> do you have to use some sort of scheduling framework on top of java?

    >
    >> Use multiple threads?

    > ~
    > a) I need to actually scan large text files (10+ million lines).
    > b) On each line there is a NL sentence.
    > c) That processing should be run only once, but as fast as possible.
    > ~
    > d) If you go:
    > d.1) int iPrx = Runtime.getRuntime().availableProcessors();
    > d.2) count all lines
    > d.3) split the file in (total lines)/iPrx
    > d.4) then run iPrx threads (or executable instances using a batch script)
    > the time you waste on d.2) and d.3) will make all that strat senseless


    How slow is the NL processing? Does it make any sense to read lines in one
    thread and pass each off to one of the iPrx-1 other threads that might run on
    separate processors?

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
     
    David Lamb, Jul 31, 2012
    #3
  4. To: qwertmonkey
    From: Joshua Cranmer <>

    [Gah, your newsreader is incapable of threading posts correctly. Please find a
    non-broken one.]

    On 7/30/2012 7:27 PM, wrote:
    >>> Is there a way to make these processors pick up/share work also, or
    >>> do you have to use some sort of scheduling framework on top of java?

    >
    >> Use multiple threads?

    > ~
    > a) I need to actually scan large text files (10+ million lines).
    > b) On each line there is a NL sentence.
    > c) That processing should be run only once, but as fast as possible.


    Only 10M-line files?

    The easiest way to do this is to just make a ThreadPoolExecutor and have your
    main thread dispatch requests as fast as possible to the pool. Or you can do
    the work pooling yourself, which may be faster since you're not continually
    posting Runnable's, but timing results would be necessary to convince me.

    There are other options, but chances are, your disk drive is going to saturate
    first (in short, it involves reading non-consecutive pages of the file, which
    is generally a recipe for disaster).

    --
    Beware of bugs in the above code; I have only proved it correct, not tried it.
    -- Donald E. Knuth

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
     
    Joshua Cranmer, Jul 31, 2012
    #4
  5. To: qwertmonkey
    From: "John B. Matthews" <>

    In article <jv759b$s7$>,
    wrote:

    > d.2) count all lines


    Maybe ask ProcessBuilder to `wc -l`, or similar?

    --
    John B. Matthews
    trashgod at gmail dot com
    <http://sites.google.com/site/drjohnbmatthews>

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
     
    John B. Matthews, Jul 31, 2012
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    225
    Roedy Green
    Jul 31, 2012
  2. Replies:
    3
    Views:
    223
    John B. Matthews
    Jul 31, 2012
  3. Replies:
    2
    Views:
    234
    Joshua Cranmer
    Jul 31, 2012
  4. Replies:
    0
    Views:
    171
  5. Replies:
    0
    Views:
    128
Loading...

Share This Page