Concurrent Write to a single file (in multi thread code)?

Discussion in 'Java' started by Donkey Hot, Apr 26, 2008.

  1. Donkey Hot

    Donkey Hot Guest

    wrote in news:785b15c5-36df-4bcf-bf49-
    :

    > Hi All,
    >
    > I am writing a program that opens a huge flat file, process it and
    > write it to another file. Just in case that I get to any performance
    > issue, I am considering implement it in multi-threading fashion.
    > I remember from old days that I need to write a factory, distributer,
    > filewriter class, locking/unlocking routine in order to achieve it.
    > Someone told me instead of all those, use non-blocking IO (NIO) and
    > ask all threads to write to a single file without thinking about
    > locking/semaphore.
    >
    > Is it true? If yes, anybody has a sample code for it? Is it easier
    > than writing it the way I've described?
    > If no, anybody have a sample of code do such a thing?
    >
    >
    > Thanks in advance,
    >
    > Homer
    >


    If playing with threads, why don't you just write one more thread, which is
    the one writing to the destination file. All other threads sent the data to
    it, which has a buffer (list maybe) for the data to write.

    No locks, one writer.
     
    Donkey Hot, Apr 26, 2008
    #1
    1. Advertising

  2. Donkey Hot

    Guest

    Hi All,

    I am writing a program that opens a huge flat file, process it and
    write it to another file. Just in case that I get to any performance
    issue, I am considering implement it in multi-threading fashion.
    I remember from old days that I need to write a factory, distributer,
    filewriter class, locking/unlocking routine in order to achieve it.
    Someone told me instead of all those, use non-blocking IO (NIO) and
    ask all threads to write to a single file without thinking about
    locking/semaphore.

    Is it true? If yes, anybody has a sample code for it? Is it easier
    than writing it the way I've described?
    If no, anybody have a sample of code do such a thing?


    Thanks in advance,

    Homer
     
    , Apr 26, 2008
    #2
    1. Advertising

  3. wrote:
    > Hi All,
    >
    > I am writing a program that opens a huge flat file, process it and
    > write it to another file. Just in case that I get to any performance
    > issue, I am considering implement it in multi-threading fashion.


    I would not do anything like this on a "just in case" basis. Most jobs
    that read a long file, process it, and write it to another file are
    limited by disk read and write performance. However, you should put the
    application together as simply as possible, and then measure it. If it
    runs fast enough, do nothing. If it is too slow, your actions should
    depend on whether the problem is I/O or CPU.

    For I/O, which is more likely to be the limiting factor, I would try to
    keep the reads and write sequential if at all possible, to minimize disk
    head movement and take advantage of OS optimizations. Also, put the
    input and output files on different disks. If the files are really huge,
    consider disk striping.

    Multi-threading might help in a couple of cases. One is if the reads are
    necessarily random, so that the disk drive has idle time while waiting
    for your program to issue the next read. However, if that is the problem
    I would begin by looking at java.nio.

    Another case for multi-threading is if the CPU time for processing the
    file is longer than the I/O time for reading it, an unusual situation.
    If so, multi-threading on suitable hardware would let you spread the
    work out over multiple processors, or multiple hardware threads in one
    processor.

    Patricia
     
    Patricia Shanahan, Apr 26, 2008
    #3
  4. look into random access files
     
    Chase Preuninger, Apr 27, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Pep
    Replies:
    6
    Views:
    826
  2. Replies:
    26
    Views:
    1,027
    John C. Bollinger
    Dec 1, 2005
  3. www
    Replies:
    13
    Views:
    627
    Patricia Shanahan
    Feb 7, 2007
  4. liu yang
    Replies:
    4
    Views:
    2,015
    Antoninus Twink
    Jul 28, 2008
  5. sam.green
    Replies:
    3
    Views:
    2,923
    jeppe
    Oct 8, 2008
Loading...

Share This Page