Re: How can you make idle processors pick up java work?

Discussion in 'Java' started by Patricia Shanahan, Aug 1, 2012.

  1. From: Patricia Shanahan <>

    On 7/31/2012 1:07 PM, qwertmonkey wrote: ...
    > Please, let me know if you have a better idea of how to read very large

    text
    > files in a faster way (while using a regular/sub-optimal box) ~

    ....

    It sounds as though your throughput is limited by disk read head time -
    splitting across physical disks increases that resource.

    You *may* be able to improve the efficiency of one drive by using NIO
    non-blocking reads to keep more disk prefetch reads going at once. Given a pool
    of reads, the drive can optimize their order to reduce total head movement, and
    therefore time the read head wastes not actually doing a transfer.

    Patricia

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
    Patricia Shanahan, Aug 1, 2012
    #1
    1. Advertising

  2. Patricia Shanahan

    Lew Guest

    To: Patricia Shanahan
    From: Lew <>

    Patricia Shanahan wrote:
    > qwertmonkey wrote:
    > ...
    >
    >> Please, let me know if you have a better idea of how
    >> to read very large text
    >> files in a faster way (while using a regular/sub-optimal box) ~

    >
    > ...


    Another approach is to farm out one file per thread, rather than multiple
    threads per file.

    Part of the game of concurrency is to figure out the right size for a chunk.

    Your problem also seems amenable to a map-reduce approach.

    > It sounds as though your throughput is limited by disk read head time -
    > splitting across physical disks increases that resource.
    >
    > You *may* be able to improve the efficiency of one drive by using NIO
    > non-blocking reads to keep more disk prefetch reads going at once. Given
    > a pool of reads, the drive can optimize their order to reduce total head
    > movement, and therefore time the read head wastes not actually doing a
    > transfer.


    You can also perhaps improve efficiency with the one thread-per-file approach.

    I/O in each thread should be bursty, especially if you use
    'BufferedReader' with large buffers. Each thread's I/O time
    is potentially CPU time for other threads. Also, multiple threads' I/O might
    bunch up for elevator seeking as Patricia describes.

    However, threads hit a limit of usefulness if you get too many. Fortunately
    that limit should be higher than that imposed by your I/O subsystem. You
    control this by setting your thread pool's maximum count and other control
    factors appropriately.

    For the kind of situation you describe I'd predict the optimum maximum to be
    somewhere around two to four times your core count.

    I don't know whether NIO or thread-based expansion will help you more. Even the
    predictions I am making are implicity heavily burdened with caveats to measure
    and take nothing for granted.

    I'd likely use a one thread-per-file approach for structural reasons, without
    much initial concern for whether it's the best performing. It's a clean
    architecture that allows good locality of working variables and a natural path
    to a service approach. I wouldn't expect any optimization beyond large buffers
    to help individual file performance, so I'd plan for an architecture that
    scales well with increased hardware. One file thread per file scales well to
    one service call per file, readily distributed across a cluster. It also fits
    well with expansions to an I/O subsystem.

    Amdahl's Law is your friend if you use it to plan units of work that have
    little to no common paths.

    --
    Lew

    --- BBBS/Li6 v4.10 Dada-1
    * Origin: Prism bbs (1:261/38)
    --- Synchronet 3.16a-Win32 NewsLink 1.98
    Time Warp of the Future BBS - telnet://time.synchro.net:24
    Lew, Aug 1, 2012
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    211
    Roedy Green
    Jul 31, 2012
  2. Replies:
    3
    Views:
    210
    John B. Matthews
    Jul 31, 2012
  3. Replies:
    2
    Views:
    215
    Joshua Cranmer
    Jul 31, 2012
  4. Replies:
    0
    Views:
    156
  5. Replies:
    0
    Views:
    114
Loading...

Share This Page