Help on a strategy for writing files of aprox. 0.5mb to disk at arate of 30 files per second

[Ben Jeurissen]

Hello,

I have to deal with the following issue in C++:

Two threads are started from the main thread, both capturing images from
a different firewire camera. Both threads take shots of 460800 bytes at
a rate of 30 frames per second. This works fine, without frame loss, and
I can display the two framestreams on screen.

Now I would like to write these frames to my disk too (2 10000rpm in
striping mode raid), but my first naive attempts (just writing the
images to disk image by image), causes the fwrite()'s to block after a
while. What appraoch is suitable for this kind of problem? Is there some
sort of "pattern" for it? Some buffering strategy which avoids those blocks?

(The images don't actually have to be saved as individual files on disk,
they can be written in lager blocks and separated later (whitout having
to worry about the 30 frames per second constraint).)

Any advice greatly appreciated.

Ben

 

[Rolf Magnus]

Hello,

I have to deal with the following issue in C++:

Two threads are started from the main thread, both capturing images
from a different firewire camera. Both threads take shots of 460800
bytes at a rate of 30 frames per second. This works fine, without
frame loss, and I can display the two framestreams on screen.

Now I would like to write these frames to my disk too (2 10000rpm in
striping mode raid), but my first naive attempts (just writing the
images to disk image by image), causes the fwrite()'s to block after a
while. What appraoch is suitable for this kind of problem? Is there
some sort of "pattern" for it? Some buffering strategy which avoids
those blocks?

(The images don't actually have to be saved as individual files on
disk, they can be written in lager blocks and separated later (whitout
having to worry about the 30 frames per second constraint).)

Any advice greatly appreciated.

Your question isn't really a C++ question (nothing language specific in
there), so comp.lang.c++ isn't the right group. The issue is very
dependant on the operating system, so I suggest asking in a newsgroup
about your OS. If you want to stream data at hight bandwidth to your
hard drive, you usually need to use special system dependant functions,
e.g. to avoid caching, which doesn't help in that case, but rather slow
things down. Memory mapped files are said to be a good candidate for
fast writing of bigger amounts of data.

 

[Phlip]

In my implementation of iostream, the writting to the output stream is
handled by the "write" function found in unistd.h. The difference
between the fwrite you used in your program & write used in C++'s
streams interface seems to be that fwrite understands the size of user
defined objects to be written to the stream. Hmm... This one looks like
you might have either a hardware problem, i.e. your RAID write
operations take too long, or your OS has some weaknesses in handling
demanding write operations.

OSs support the ability to queue a batch of commands to the disk driver
directly. This is how, for example, IIS or SQL Server blast bits hither and
yon without waiting for our feeb C++ library code to catch up.

Ben Jeurissen wrote:

OSs also support the ability to multiplex IO together at a single port. If
the purpose of the two threads is so each can wait for input and then
efficiently dispatch it, then threads are overkill here. The batches of
commands I mentioned also have the ability to run asynchronously, so your
architecture could instead use one thread, collect input and add it to
batches, and ship out the batches when they fill up. This would narrow your
code to the most efficient activities, without using hacks like threads or C
library functions to squeeze performance out of systems that are not
optimized for throughput as high as yours.

 

[Evan Carew]

Ben,

In my implementation of iostream, the writting to the output stream is
handled by the "write" function found in unistd.h. The difference
between the fwrite you used in your program & write used in C++'s
streams interface seems to be that fwrite understands the size of user
defined objects to be written to the stream. Hmm... This one looks like
you might have either a hardware problem, i.e. your RAID write
operations take too long, or your OS has some weaknesses in handling
demanding write operations.

Evan Carew

 

[Thomas Matthews]

Hello,

I have to deal with the following issue in C++:

Two threads are started from the main thread, both capturing images from
a different firewire camera. Both threads take shots of 460800 bytes at
a rate of 30 frames per second. This works fine, without frame loss, and
I can display the two framestreams on screen.

Now I would like to write these frames to my disk too (2 10000rpm in
striping mode raid), but my first naive attempts (just writing the
images to disk image by image), causes the fwrite()'s to block after a
while. What appraoch is suitable for this kind of problem? Is there some
sort of "pattern" for it? Some buffering strategy which avoids those
blocks?

(The images don't actually have to be saved as individual files on disk,
they can be written in lager blocks and separated later (whitout having
to worry about the 30 frames per second constraint).)

Any advice greatly appreciated.

Ben

Since you didn't specify the platform, I suggest you peruse over
at or maybe something under
which suits your platform better.

The underlying issue is a real-time speed issue. One component
is slower than another. The pattern to resolve this is called
"double buffering". One task fills a buffer while another
empties it. Sometimes, more than one buffer may be necessary
to resolve the speed issues. Many OSes use this technique
when "spooling" files to the printer.

In your case, see if your disk controller has any intelligence
or DMA available. The best idea is for you to give the disk
controller the start address and quantity of data, then let
it go. Have it interrupt you when it is finished. Let the
interrupt handler continually feed the disk controller when
there are buffers ready.

Now the C++ part (or perhaps getting closer). I suggest a
circular queue (a.k.a. ring buffer) of pointers to buffers.
Let the input task fill buffers then insert them into the
queue. The Disk task will feed the disk controller then
remove the pointers from the queue. Don't allocate any
memory during this process; all buffers should be allocated
during the initialization of your program.

--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book

 

[kal]

Ben Jeurissen wrote:

Now the C++ part (or perhaps getting closer). I suggest a
circular queue (a.k.a. ring buffer) of pointers to buffers.
Let the input task fill buffers then insert them into the
queue. The Disk task will feed the disk controller then
remove the pointers from the queue. Don't allocate any
memory during this process; all buffers should be allocated
during the initialization of your program.

The pre-allocation of buffers is a nice idea. You can also pre-allocate
the disk file in which you store the images.

One really needs more than 2-threads. The thing that needs study here
is the availability of resources. Hope the following is of some use to you.

1. You can use asynchronous IO (called OVERLAPPED IO in some systems).
But these things have large overheads and are better avoided.

2. You should develop two objects, one for camera control and image
capture and the other for storage management (writing to and reading from
disks.) Each of these objects, once instantiated, should run in their own
threads. This way you will avoid the problem of having to deal with
ASYNCHRONOUS IO.

3. Pass the image data from the camera module to the storage management
module so that the later can write it to disk at its own pace. You can get
fancy here and simply pass an image OBJECT instead of doing memory copying.
But then the modules should co-operatively manage creation and destruction
of such objects.

3. Generalize your program to handle one or more cameras and one or more
associated storage management objects.

4. Generalize your program so you can vary the image resolution and hence
the image size.

5. Run the program for 2 cameras and vary the image size, say starting
from 16x16 and doubling the image size in each iteration. The image size
at which you start having problems will give you some indication of the
capacity of your machine.

6. If you need larger image sizes and/or color depths then buy a faster
machine or one with more processors.

7. You can get fancy with the storage module by using DIRECT IO, image
compression (MPEG) etc. or shove the data out to some other machine ...

Good Luck!