Help on a strategy for writing files of aprox. 0.5mb to disk at arate of 30 files per second

Discussion in 'C++' started by Ben Jeurissen, Jan 29, 2004.

  1. Hello,

    I have to deal with the following issue in C++:

    Two threads are started from the main thread, both capturing images from
    a different firewire camera. Both threads take shots of 460800 bytes at
    a rate of 30 frames per second. This works fine, without frame loss, and
    I can display the two framestreams on screen.

    Now I would like to write these frames to my disk too (2 10000rpm in
    striping mode raid), but my first naive attempts (just writing the
    images to disk image by image), causes the fwrite()'s to block after a
    while. What appraoch is suitable for this kind of problem? Is there some
    sort of "pattern" for it? Some buffering strategy which avoids those blocks?

    (The images don't actually have to be saved as individual files on disk,
    they can be written in lager blocks and separated later (whitout having
    to worry about the 30 frames per second constraint).)

    Any advice greatly appreciated.

    Ben
     
    Ben Jeurissen, Jan 29, 2004
    #1
    1. Advertising

  2. Ben Jeurissen

    Rolf Magnus Guest

    Re: Help on a strategy for writing files of aprox. 0.5mb to disk at a rate of 30 files per second

    Ben Jeurissen wrote:

    > Hello,
    >
    > I have to deal with the following issue in C++:
    >
    > Two threads are started from the main thread, both capturing images
    > from a different firewire camera. Both threads take shots of 460800
    > bytes at a rate of 30 frames per second. This works fine, without
    > frame loss, and I can display the two framestreams on screen.
    >
    > Now I would like to write these frames to my disk too (2 10000rpm in
    > striping mode raid), but my first naive attempts (just writing the
    > images to disk image by image), causes the fwrite()'s to block after a
    > while. What appraoch is suitable for this kind of problem? Is there
    > some sort of "pattern" for it? Some buffering strategy which avoids
    > those blocks?
    >
    > (The images don't actually have to be saved as individual files on
    > disk, they can be written in lager blocks and separated later (whitout
    > having to worry about the 30 frames per second constraint).)
    >
    > Any advice greatly appreciated.


    Your question isn't really a C++ question (nothing language specific in
    there), so comp.lang.c++ isn't the right group. The issue is very
    dependant on the operating system, so I suggest asking in a newsgroup
    about your OS. If you want to stream data at hight bandwidth to your
    hard drive, you usually need to use special system dependant functions,
    e.g. to avoid caching, which doesn't help in that case, but rather slow
    things down. Memory mapped files are said to be a good candidate for
    fast writing of bigger amounts of data.
     
    Rolf Magnus, Jan 29, 2004
    #2
    1. Advertising

  3. Ben Jeurissen

    Phlip Guest

    Re: Help on a strategy for writing files of aprox. 0.5mb to disk at a rate of 30 files per second

    Evan Carew wrote:

    > In my implementation of iostream, the writting to the output stream is
    > handled by the "write" function found in unistd.h. The difference
    > between the fwrite you used in your program & write used in C++'s
    > streams interface seems to be that fwrite understands the size of user
    > defined objects to be written to the stream. Hmm... This one looks like
    > you might have either a hardware problem, i.e. your RAID write
    > operations take too long, or your OS has some weaknesses in handling
    > demanding write operations.


    OSs support the ability to queue a batch of commands to the disk driver
    directly. This is how, for example, IIS or SQL Server blast bits hither and
    yon without waiting for our feeb C++ library code to catch up.

    > Ben Jeurissen wrote:


    > > I have to deal with the following issue in C++:
    > >
    > > Two threads are started from the main thread, both capturing images from
    > > a different firewire camera.


    OSs also support the ability to multiplex IO together at a single port. If
    the purpose of the two threads is so each can wait for input and then
    efficiently dispatch it, then threads are overkill here. The batches of
    commands I mentioned also have the ability to run asynchronously, so your
    architecture could instead use one thread, collect input and add it to
    batches, and ship out the batches when they fill up. This would narrow your
    code to the most efficient activities, without using hacks like threads or C
    library functions to squeeze performance out of systems that are not
    optimized for throughput as high as yours.

    --
    Phlip
    http://www.xpsd.org/cgi-bin/wiki?TestFirstUserInterfaces
     
    Phlip, Feb 2, 2004
    #3
  4. Ben Jeurissen

    Evan Carew Guest

    Re: Help on a strategy for writing files of aprox. 0.5mb to diskat a rate of 30 files per second

    Ben,

    In my implementation of iostream, the writting to the output stream is
    handled by the "write" function found in unistd.h. The difference
    between the fwrite you used in your program & write used in C++'s
    streams interface seems to be that fwrite understands the size of user
    defined objects to be written to the stream. Hmm... This one looks like
    you might have either a hardware problem, i.e. your RAID write
    operations take too long, or your OS has some weaknesses in handling
    demanding write operations.

    Evan Carew

    Ben Jeurissen wrote:
    > Hello,
    >
    > I have to deal with the following issue in C++:
    >
    > Two threads are started from the main thread, both capturing images from
    > a different firewire camera. Both threads take shots of 460800 bytes at
    > a rate of 30 frames per second. This works fine, without frame loss, and
    > I can display the two framestreams on screen.
    >
    > Now I would like to write these frames to my disk too (2 10000rpm in
    > striping mode raid), but my first naive attempts (just writing the
    > images to disk image by image), causes the fwrite()'s to block after a
    > while. What appraoch is suitable for this kind of problem? Is there some
    > sort of "pattern" for it? Some buffering strategy which avoids those
    > blocks?
    >
    > (The images don't actually have to be saved as individual files on disk,
    > they can be written in lager blocks and separated later (whitout having
    > to worry about the 30 frames per second constraint).)
    >
    > Any advice greatly appreciated.
    >
    > Ben
     
    Evan Carew, Feb 2, 2004
    #4
  5. Re: Help on a strategy for writing files of aprox. 0.5mb to diskat a rate of 30 files per second

    Ben Jeurissen wrote:

    > Hello,
    >
    > I have to deal with the following issue in C++:
    >
    > Two threads are started from the main thread, both capturing images from
    > a different firewire camera. Both threads take shots of 460800 bytes at
    > a rate of 30 frames per second. This works fine, without frame loss, and
    > I can display the two framestreams on screen.
    >
    > Now I would like to write these frames to my disk too (2 10000rpm in
    > striping mode raid), but my first naive attempts (just writing the
    > images to disk image by image), causes the fwrite()'s to block after a
    > while. What appraoch is suitable for this kind of problem? Is there some
    > sort of "pattern" for it? Some buffering strategy which avoids those
    > blocks?
    >
    > (The images don't actually have to be saved as individual files on disk,
    > they can be written in lager blocks and separated later (whitout having
    > to worry about the 30 frames per second constraint).)
    >
    > Any advice greatly appreciated.
    >
    > Ben


    Since you didn't specify the platform, I suggest you peruse over
    at news:comp.arch.embedded or maybe something under
    news:comp.arch.* which suits your platform better.

    The underlying issue is a real-time speed issue. One component
    is slower than another. The pattern to resolve this is called
    "double buffering". One task fills a buffer while another
    empties it. Sometimes, more than one buffer may be necessary
    to resolve the speed issues. Many OSes use this technique
    when "spooling" files to the printer.

    In your case, see if your disk controller has any intelligence
    or DMA available. The best idea is for you to give the disk
    controller the start address and quantity of data, then let
    it go. Have it interrupt you when it is finished. Let the
    interrupt handler continually feed the disk controller when
    there are buffers ready.

    Now the C++ part (or perhaps getting closer). I suggest a
    circular queue (a.k.a. ring buffer) of pointers to buffers.
    Let the input task fill buffers then insert them into the
    queue. The Disk task will feed the disk controller then
    remove the pointers from the queue. Don't allocate any
    memory during this process; all buffers should be allocated
    during the initialization of your program.

    --
    Thomas Matthews

    C++ newsgroup welcome message:
    http://www.slack.net/~shiva/welcome.txt
    C++ Faq: http://www.parashift.com/c -faq-lite
    C Faq: http://www.eskimo.com/~scs/c-faq/top.html
    alt.comp.lang.learn.c-c++ faq:
    http://www.raos.demon.uk/acllc-c /faq.html
    Other sites:
    http://www.josuttis.com -- C++ STL Library book
     
    Thomas Matthews, Feb 2, 2004
    #5
  6. Ben Jeurissen

    kal Guest

    Re: Help on a strategy for writing files of aprox. 0.5mb to disk at a rate of 30 files per second

    Thomas Matthews <> wrote in message news:<NizTb.2897$>...

    > Ben Jeurissen wrote:


    <...>

    > > Two threads are started from the main thread, both capturing images from
    > > a different firewire camera. Both threads take shots of 460800 bytes at
    > > a rate of 30 frames per second. This works fine, without frame loss, and
    > > I can display the two framestreams on screen.
    > >


    <...>

    > Now the C++ part (or perhaps getting closer). I suggest a
    > circular queue (a.k.a. ring buffer) of pointers to buffers.
    > Let the input task fill buffers then insert them into the
    > queue. The Disk task will feed the disk controller then
    > remove the pointers from the queue. Don't allocate any
    > memory during this process; all buffers should be allocated
    > during the initialization of your program.
    >
    > --
    > Thomas Matthews


    The pre-allocation of buffers is a nice idea. You can also pre-allocate
    the disk file in which you store the images.

    One really needs more than 2-threads. The thing that needs study here
    is the availability of resources. Hope the following is of some use to you.

    1. You can use asynchronous IO (called OVERLAPPED IO in some systems).
    But these things have large overheads and are better avoided.

    2. You should develop two objects, one for camera control and image
    capture and the other for storage management (writing to and reading from
    disks.) Each of these objects, once instantiated, should run in their own
    threads. This way you will avoid the problem of having to deal with
    ASYNCHRONOUS IO.

    3. Pass the image data from the camera module to the storage management
    module so that the later can write it to disk at its own pace. You can get
    fancy here and simply pass an image OBJECT instead of doing memory copying.
    But then the modules should co-operatively manage creation and destruction
    of such objects.

    3. Generalize your program to handle one or more cameras and one or more
    associated storage management objects.

    4. Generalize your program so you can vary the image resolution and hence
    the image size.

    5. Run the program for 2 cameras and vary the image size, say starting
    from 16x16 and doubling the image size in each iteration. The image size
    at which you start having problems will give you some indication of the
    capacity of your machine.

    6. If you need larger image sizes and/or color depths then buy a faster
    machine or one with more processors.

    7. You can get fancy with the storage module by using DIRECT IO, image
    compression (MPEG) etc. or shove the data out to some other machine ...

    Good Luck!

    -------

    "There is no problem, however complicated, which when looked at it
    the right way did not become still more complicated." -- ?
     
    kal, Feb 3, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jas Shultz
    Replies:
    0
    Views:
    987
    Jas Shultz
    Dec 3, 2003
  2. =?Utf-8?B?S01aX3N0YXRl?=

    Quick one - Is SESSION per browser instance or per IP Address?

    =?Utf-8?B?S01aX3N0YXRl?=, Apr 4, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    5,930
    gerry
    Apr 10, 2006
  3. Razvan
    Replies:
    1
    Views:
    430
    tony vee
    Sep 10, 2004
  4. Replies:
    12
    Views:
    535
    santosh
    Nov 15, 2006
  5. simonbun
    Replies:
    4
    Views:
    492
    thebjorn
    Jul 29, 2007
Loading...

Share This Page