Concurrent Write to a single file (in multi thread code)?

[Donkey Hot]

(e-mail address removed) wrote in (e-mail address removed):

Hi All,

I am writing a program that opens a huge flat file, process it and
write it to another file. Just in case that I get to any performance
issue, I am considering implement it in multi-threading fashion.
I remember from old days that I need to write a factory, distributer,
filewriter class, locking/unlocking routine in order to achieve it.
Someone told me instead of all those, use non-blocking IO (NIO) and
ask all threads to write to a single file without thinking about
locking/semaphore.

Is it true? If yes, anybody has a sample code for it? Is it easier
than writing it the way I've described?
If no, anybody have a sample of code do such a thing?


Thanks in advance,

Homer

If playing with threads, why don't you just write one more thread, which is
the one writing to the destination file. All other threads sent the data to
it, which has a buffer (list maybe) for the data to write.

No locks, one writer.

 

[homicanada]

Hi All,

I am writing a program that opens a huge flat file, process it and
write it to another file. Just in case that I get to any performance
issue, I am considering implement it in multi-threading fashion.
I remember from old days that I need to write a factory, distributer,
filewriter class, locking/unlocking routine in order to achieve it.
Someone told me instead of all those, use non-blocking IO (NIO) and
ask all threads to write to a single file without thinking about
locking/semaphore.

Is it true? If yes, anybody has a sample code for it? Is it easier
than writing it the way I've described?
If no, anybody have a sample of code do such a thing?


Thanks in advance,

Homer

 

[Patricia Shanahan]

Hi All,

I am writing a program that opens a huge flat file, process it and
write it to another file. Just in case that I get to any performance
issue, I am considering implement it in multi-threading fashion.

I would not do anything like this on a "just in case" basis. Most jobs
that read a long file, process it, and write it to another file are
limited by disk read and write performance. However, you should put the
application together as simply as possible, and then measure it. If it
runs fast enough, do nothing. If it is too slow, your actions should
depend on whether the problem is I/O or CPU.

For I/O, which is more likely to be the limiting factor, I would try to
keep the reads and write sequential if at all possible, to minimize disk
head movement and take advantage of OS optimizations. Also, put the
input and output files on different disks. If the files are really huge,
consider disk striping.

Multi-threading might help in a couple of cases. One is if the reads are
necessarily random, so that the disk drive has idle time while waiting
for your program to issue the next read. However, if that is the problem
I would begin by looking at java.nio.

Another case for multi-threading is if the CPU time for processing the
file is longer than the I/O time for reading it, an unusual situation.
If so, multi-threading on suitable hardware would let you spread the
work out over multiple processors, or multiple hardware threads in one
processor.

Patricia

 

[Chase Preuninger]

look into random access files