java thread for core 2 due processors

Discussion in 'Java' started by Motaz K. Saad, Aug 17, 2009.

  1. Hello,

    I am working on file processing (large number of files) which is
    totally independent task for each file. I want to speedup processing
    using java thread but I do not want overwhelm my processor (core 2 due
    processor) with all threads at the same time.

    I want to run 2 parallel threads simultaneously and wait until they
    finish then run the next 2 threads. furthermore, each thread allocate
    large amount of memory so I need to de-allocate the thread after
    collecting the result from it.

    I would appreciate if anyone direct my to similar set example
    I would appreciate guideline and help


    Thanks,
    --
    Motaz K. Saad
    Motaz K. Saad, Aug 17, 2009
    #1
    1. Advertising

  2. Motaz K. Saad

    Tom Anderson Guest

    On Mon, 17 Aug 2009, Motaz K. Saad wrote:

    > I am working on file processing (large number of files) which is totally
    > independent task for each file. I want to speedup processing using java
    > thread but I do not want overwhelm my processor (core 2 due processor)
    > with all threads at the same time.


    You'd have to have an awful lot of threads to 'overwhelm' the processor -
    hundreds, or perhaps even thousands. There is a chance that it might
    overwhelm the disk, though, depending on things like how big your files
    are and how the threads access them.

    However, running your jobs in 1000 threads won't be any faster than
    running them in 10 (although that might be faster than running them in 2,
    because it lets you saturate the disk - two threads on two CPUs will leave
    either the disk or CPU underutilised at some point, unless readahead and
    GC keep them busy), so your conclusion is right - you want to use fewer
    threads than you have tasks.

    > I want to run 2 parallel threads simultaneously and wait until they
    > finish then run the next 2 threads.


    No, you want to run two parallel tasks simultaneously and wait until they
    finish then run the next two tasks. You don't need to have one thread per
    task.

    > furthermore, each thread allocate large amount of memory so I need to
    > de-allocate the thread after collecting the result from it.


    As long as there are no pointers to the allocated objects after the file
    is processed, the garbage collector will reclaim it.

    > I would appreciate if anyone direct my to similar set example I would
    > appreciate guideline and help


    You want:

    http://java.sun.com/javase/6/docs/api/java/util/concurrent/ExecutorService.html

    Your code looks like:

    public class FileProcessingTask implements Runnable {
    private final File file;
    public FileProcessingTask(File file) {
    this.file = file;
    }
    public void run() {
    try {
    // process file
    }
    catch (Exception e) {
    // log the exception
    // close any open files
    }
    }
    }

    public class FileProcessingApp {
    public static void main(String... args) {
    Collection<File> filesToProcess; // initialise this however you like
    int numThreads = Runtime.getRuntime().availableProcessors() * 2;
    ExecutorService executor = Executors.newFixedThreadPool(numThreads);
    for (File file: filesToProcess) {
    executor.execute(new FileProcessingTask(file));
    }
    executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    }
    }

    tom

    --
    Brace yourself for an engulfing, cowardly autotroph! I want your
    photosynthetic apparatii!
    Tom Anderson, Aug 17, 2009
    #2
    1. Advertising

  3. Motaz K. Saad wrote:
    > I am working on file processing (large number of files) which is
    > totally independent task for each file. I want to speedup processing
    > using java thread but I do not want overwhelm my processor (core 2 due
    > processor) with all threads at the same time.


    Depending on what you are exactly doing, it seems to me that doing two
    threads at a time will not be maximizing throughput. They could be
    spending most of their time waiting around for disk I/O. On modern
    processors, one thread-per-core is generally not optimal; ISTR hearing
    somewhere that the correct number is about 1.5-2 per core.

    > I want to run 2 parallel threads simultaneously and wait until they
    > finish then run the next 2 threads. furthermore, each thread allocate
    > large amount of memory so I need to de-allocate the thread after
    > collecting the result from it.


    As long as you watch who holds references to what, the GC will clear
    memory up itself. I think it would generally be sufficient to make sure
    that you don't keep references to the threads unless necessary--good
    programming practices will likely confine the leaks.

    > I would appreciate if anyone direct my to similar set example
    > I would appreciate guideline and help


    The first thing to recommend is that you have a grasp on how to do
    concurrent programming. If your tasks do not communicate to each other
    or to the main program, you probably don't have any thread-safety issues.

    The best way to actually implement this is probably with the new from
    Java 5 ExecutorService API, as others have stated.

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
    Joshua Cranmer, Aug 17, 2009
    #3
  4. Motaz K. Saad

    Roedy Green Guest

    On Mon, 17 Aug 2009 19:16:06 +0100, Tom Anderson
    <> wrote, quoted or indirectly quoted someone who
    said :

    >
    >However, running your jobs in 1000 threads won't be any faster than
    >running them in 10 (although that might be faster than running them in 2,


    The main time large numbers of threads buys you something is web
    scraping. Your threads are mostly sitting waiting for socket i/o to
    complete.
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.
    Roedy Green, Aug 18, 2009
    #4
  5. Motaz K. Saad

    Roedy Green Guest

    On Tue, 18 Aug 2009 15:42:03 +0200, Christian <> wrote,
    quoted or indirectly quoted someone who said :

    >isn't that what NIO is for so you need only one thread to read.. and he
    >gives off the processing to an ExecutorService..


    Yes. I've been told that is a less flexible, more efficient way to do
    it.
    --
    Roedy Green Canadian Mind Products
    http://mindprod.com

    http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.
    Roedy Green, Aug 21, 2009
    #5
  6. Motaz K. Saad

    Tom Anderson Guest

    On Sat, 22 Aug 2009, Christian wrote:

    > so I assume that if you really have no interaction with the os and
    > nothing that could block you. Then number of Threads = CPU cores should
    > be fine/perfect.


    Close.

    If you're making garbage, you actually want slightly fewer threads than
    cores, because the collector will need some CPU time every so often. If
    you have 4 cores, you want to run about 3.8 threads. Of course, this is
    not possible in practice.

    Also, bear in mind that 'no interaction with the OS' includes 'not using
    any virtual memory' (in the sense of 'memory which is paged out') - a
    thread which accesses a memory location which is paged out will trigger
    disk IO, which will cause it to block. If your java processes fits
    entirely in physical memory, this is not an issue.

    tom

    --
    Information is not knowledge. -- Albert Einstein
    Tom Anderson, Aug 25, 2009
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    1
    Views:
    910
    Kai Schwebke
    Mar 14, 2007
  2. rym3k
    Replies:
    1
    Views:
    951
    =?UTF-8?B?QXJuZSBWYWpow7hq?=
    Jul 6, 2007
  3. Sanny
    Replies:
    29
    Views:
    975
    Twisted
    Jul 25, 2007
  4. Replies:
    1
    Views:
    298
    Ian Collins
    Aug 20, 2007
  5. Fresh
    Replies:
    2
    Views:
    622
    Bo Persson
    Apr 22, 2008
Loading...

Share This Page