Memory Mapped IO and Sample code for win2000

Discussion in 'C++' started by Peter Nolan, Oct 8, 2004.

  1. Peter Nolan

    Peter Nolan Guest

    Hi All,
    I have some software that currently loads database tables into sorted arrays
    in memory to be binary searched by processes. (Long story as to why it does
    this..)

    Each process must either read the data from the in memory array or from the
    database itself if the table is too large to be loaded into memory. (2GB is
    usually the limit because the array is indexed by a 32 bit signed integer.
    I have never had one of these tables go over 2GB, even when the client had
    20M customers.)

    This current set up is fine for companies with a few million customers and a
    few million transactions per day. The software works just fine on win2000
    and solaris/aix. None of my existing customers has large enough lookup
    tables or so many transactions that it is a problem to load them into
    memory.

    However, one of the major 'upgrades' I want to make to this code is to
    enable it to share these in-memory arrays between many processes. This
    sharing of memory will make the product pretty much 'endlessly scalable',
    and this is where I want to get it to.

    The idea is that if the customer is a large telephone company then I can
    split the Call Detail Records into say 12 input files and have 12 processes
    run at the same time. By sharing the tables to be searched in memory I can
    cut the memory requirement by a factor of 12. By putting all the data to be
    looked up into memory the speed of processing the CDRs can be maximised
    because a lookup to a database is at least 20 times slower than the lookup
    to the array even if the data is in memory.

    I have the code I need to do this under solaris.....but from the examples I
    can see in VS.NET it looks like windows 2000 acts a little differently.

    Questions....

    Q1. Does anyone have a fragement of code for a producer/consumer for memory
    mapped IO under win2000 that they are willing to share? Or is there a good
    public example anywhere that someone here can point to? The examples I can
    find with vs.net are in java or very old.

    Q2. Is there a 2GB limit on creating a memory mapped file under win2000?

    Q3. Will a memory mapped file like this run at the same speed as an in
    memory array that is created by calloc like the current one? (Once the
    producer has loaded the table into memory all other processes are
    'read-only' and will be able to read the memory at the same time, no locks
    required.)

    Q4. Are there any other pointers the experts on this group might like to
    pass along to someone going to do their very first Memory Mapped IO piece of
    work? Are there any 'gotchas' for beginners in this area?

    All assistance is much appreciated!!!

    Best Regards

    Peter Nolan
    www.peternolan.com
     
    Peter Nolan, Oct 8, 2004
    #1
    1. Advertising

  2. Peter Nolan wrote:

    > Hi All,
    > I have some software that currently loads database tables into sorted
    > arrays in memory to be binary searched by processes. (Long story as to why
    > it does this..)

    ... deleted
    > All assistance is much appreciated!!!
    >
    > Best Regards
    >
    > Peter Nolan
    > www.peternolan.com


    Hi, the ACE framework (http://www.cs.wustl.edu/~schmidt/ACE.html) provides
    memory mapped files on all it's supported platforms (man unices, embedded
    systems and win32)

    Lothar
     
    Lothar Werzinger, Oct 8, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Evan Carew

    Re: B+Tree and Memory Mapped Files

    Evan Carew, Apr 1, 2004, in forum: C++
    Replies:
    0
    Views:
    671
    Evan Carew
    Apr 1, 2004
  2. Thomas Matthews

    Re: B+Tree and Memory Mapped Files

    Thomas Matthews, Apr 2, 2004, in forum: C++
    Replies:
    2
    Views:
    786
    Kevin Goodsell
    Apr 2, 2004
  3. Michael Schwarz

    Memory Usage ASP.NET (Win2000)

    Michael Schwarz, Nov 13, 2006, in forum: ASP .Net
    Replies:
    2
    Views:
    323
    Cowboy \(Gregory A. Beamer\)
    Nov 13, 2006
  4. Replies:
    3
    Views:
    591
    Dietmar Kuehl
    Feb 23, 2006
  5. Luki
    Replies:
    1
    Views:
    90
Loading...

Share This Page