How can I create a Ram Disk within C/C++ program?

[Kenny McCormack]

Well, if you're running it on a Linux system, it's not utterly useless.

But in the religion of this group, if it isn't 120% guaranteed to work
on not only every machine/platform in existence, but also every single
machine/platform conceivable by the human mind, then it ain't worth
spitting on... (but that doesn't stop them from doing so)

 

[Frederick Gotham]

Tom posted:

I'd greatly appreciate advice and code snippets on how to create a ram
disk within a C/C++ program.

I also need to be able to determine the free space.

Thanks in advance for any help.

struct RamDisk { int i; }

int main(void)
{
struct RamDisk obj;
return 0;
}

 

[Tom]

Call it sarcasm, or a joke, or perhaps just taking things too literally.

I'm still not sure whether Tom's reply was being sarcastic in return or
if he was really thanking me without realising that the code was useless
to him.

I "Truly" *was* and still am thanking you. In my opinion sarcasm is an
inefficient way of teaching people. Certainly among experts sarcasm at
the level suggested could catch you just right and be gut busting
funny; however, expert I will never be. I think you were being helpful
and provided a perfect solution for the environment which you use.
Certainly illustrates a powerful feature within Linux.

In my ignorance I did not initially define which operating system or C
compiler I was using. Your reply appears to me as magical because I
had never seen the "system" commands used before. I have never used
Linux. Without the additional comments stating how non-portable your
solution was ... I would have been trying to get it to work on my Win
2000 set-up. To think that my knowledge is so limited that I would
have actually attempted this ... really is funny!

Other bits that further illuminate my ignorance are >>

The usage of:

long long space = create_ramdisk();

I am guessing you are "naming" or creating a "space" of some sort
within the program. I have read in the past about "named" spaces but
that topic really went over my head at that time and remains there
today. Until now ... I have been able to work around not understanding
named spaces but I should revisit that subject soon. Perhaps now it
will sink in.

Another bit that caught my eye was the "%lld" usage. I have used "%ld"
many times but have never even seen double precision specified in such
manner.

At this time I need to pursue whatever it takes to make this Ram disk
feature work in Win 2000. One Win 2000 specific article I have found
is the following:

http://support.microsoft.com/kb/q257405/

This task might become hairball nasty difficult. I really don't know.
Any suggestions on good books that bridge between C code and beginner
level system programming?

Comparing the solution for a Win 2000 system to that of a Linux
solution will be most interesting and educational for me. Using a Ram
disk for security reasons (Kiosk) or harsh environments (high
vibration) is very interesting. Even if it is not the best solution
for my specific task I am learning a little outside the box.

Some have suggested profiling to determine the exact bottlenecks. I
have no profiling experience but understand a slow I/O task made fast
using a Ram disk or other technique may not speed up the program any
what-so-ever if it was not previously the rate limiting step ... and
if the I/O task is truly performed parallel to the other tasks.
However, if the I/O has any sequential behavior ... even if it is not
the longest duration event ... saving time on that specific
sequentially acting task speeds the program. It seems to me that a Ram
disk would make read time as fast as possible and thus making read
time no longer a consideration for speed improvements. However, would
not cache management still be handling the I/O from the Ram disk? So
another method of some sort might be a better solution? A method that
simply retrieves values designated as in memory and entirely avoids
any internal cache management logic? After all, why should cache
memory be used to hold something already residing in memory? Even if
the time penalty is insignificant ... it is still double handling the
data.

My original attempts at specifying a very large array for this task
failed. Anytime I exceeded approximately 600,000 for the array size
.... the program would not compile.

Quoting from MSDN Library >>
----------------------------
Largest Array Size
ANSI 3.3.3.4, 4.1.1 The type of integer required to hold the maximum
size of an array—that is, the size of size_t

The size_t typedef is an unsigned int with the range 0x00000000 to
0x7CFFFFFF.
----------------------------

The above range yeilds a maximum indexing value of 2,097,151,999. Am I
misreading the doco somehow?

Some have suggested some type of memory mapping technique. I have done
some searching in that area and have not yet found a technical reason
for memory mapping to be of benefit. Its name certainly sounds
promissing ... so I will continue searching in that area too.

Boost provides some library routines too. I have not used any Boost
routines as of yet; however, the boost.filesystem provides some
interesting functionality ... but still does not manage the memory
space as I am hoping to do.

I'm confused how any code that is "entirely" portable can be
considered to be useless?

 

[Tom]

Tom posted:



struct RamDisk { int i; }

int main(void)
{
struct RamDisk obj;
return 0;
}

Fred -- I fail to see how a structure that shares the name of the task
I need to perform and has a single integer for a member can manage a
relatively large block of data from a huge data file.

If you want to populate that structure with a few doubles and ints ...
then try it as an array of size 1,000,000 or so ...

struct RamDisk obj[1000000];

.... then you'd be right back where my compiler has already choked.

 

[Chris Torek]

[RAM-disk creation and manipulation is] no more difficult and no
less system specific than it was back then.

Well, no less system-specific, to be sure. Sometimes "more (or
less) difficult" depending on the system. (Aside: in vxWorks, just
include the "ram disk" component in your project. The default name
is "/ram0" but you can change it. If you want more than one, it is
somewhat more difficult.)

Indeed.

It still makes sense on diskless clients that boot over NFS and so on.
It's also indicated in some applications where security is sensitive, or
for devices that need to operate in environments where moving parts are
impractical.

Sometimes, a physical (as opposed to virtual, in-kernel-only) RAM
disk is appropriate. That is, you have a piece of hardware you
plug in, which looks and acts like a disk as far as the system is
concerned, but actually stores data in RAM.

While many systems (including vxWorks, as mentioned above) do
include in-main-memory pseudo-disk devices, in a purely theoretical
sense, there is *never* [%] any reason to use one: whatever you
are doing that is going through some sort of "file system" layer,
only to wind up going directly to regular old ordinary memory, is
going through a "unnecessary" manipulation. File systems do a lot
of work in order to deal with structured, and somewhat klunky, disk
drive hardware, where memory exists in the form of "disk blocks" or
"sectors" that can only be rewritten in entire units, cannot be
moved or resized, and is generally very slow to access (about 5 or
6 decimal orders of magnitude slower than RAM). Real RAM does not
share most of these characteristic drawbacks, and going through
a (usually quite heavy) software layer that simply pretends to add
some of them back is just silly.

[% One obvious exception is when the RAM-disk is being used for
testing the file system software. Here, even in theory, the RAM
drive is useful. ]

In practice, the reason for using a RAM disk is that the stubborn
software (written by some stubborn and/or or long-gone programmer)
insists on using the file system interface, and the system provides
no way to short-circuit this. For instance, in C -- to get at
least marginally back on topic -- one might have a program (or
large library) that does all its I/O to a "stdio" stream. Since
ISO C has no way to "open memory as a stream", one is often forced
to supply an actual, on-disk file. To get decent performance, one
may wish to use an "on-RAM-disk file" instead of "on-actual-disk".

Note that it can make sense to use a "RAM file system" rather than
a "RAM disk" in this case. Going through a *real* file system,
you may do a whole lot of extra work trying to avoid talking too
much to a slow (real) disk -- i.e., spend large amounts of CPU time
to avoid waiting for the device. But if the "device" is fake, and
actually is very fast (relative to real disks), this CPU time is
being wasted. It may be better to go through a fake file system
(one which uses RAM), instead of a real file system on a fake disk.

Of course, it would be even better still if C had some sort of
memory-oriented stdio streams -- or something like the 4.4BSD
"funopen", which is even more general. (I am pretty sure that
funopen() did get proposed for C99; but we might consider ourselves
lucky to have gotten even snprintf(). )

 

[Nick Keighley]

I'm confused how any code that is "entirely" portable can be
considered to be useless?

Kenny McCormack makes silly remarks on comp.lang.c just to get a
reaction. This is known as "trolling"; it is best to ignore him.

<snip>

 

[bali]

The particular semantics of a "ram disk" depend entirely on how your
operating system implements filesystems and/or storage devices.

I could go into great detail of how ram disks are implemented in C, in
the linux driver, but it would be way off topic for comp.lang.c.

Do you actually want a RAM disk, or do you merely want a memory
structure and access methods that work like a RAM disk but are local to
your program? Maybe what you really want is an in-memory database.

Or maybe you really do want to write a RAM disk. In that case you'd
better decide what platform you're using and ask in a forum appropriate
to it.

Somewhere, I have example code that includes a Windows NT RAM Disk
driver which is implemented in C++ and inline ASM.

 

[Frederick Gotham]

Tom posted:

Fred -- I fail to see how a structure that shares the name of the task
I need to perform and has a single integer for a member can manage a
relatively large block of data from a huge data file.

As I. I also fail to see how ramdisks would be topical on comp.lang.c.

If you want to allocate a load of memory, then use "malloc". If you want to
give it a drive letter, then go to a newsgroup which discusses your platform.

 

[Tom]

[RAM-disk creation and manipulation is] no more difficult and no
less system specific than it was back then.

Well, no less system-specific, to be sure. Sometimes "more (or
less) difficult" depending on the system. (Aside: in vxWorks, just
include the "ram disk" component in your project. The default name
is "/ram0" but you can change it. If you want more than one, it is
somewhat more difficult.)

Indeed.

It still makes sense on diskless clients that boot over NFS and so on.
It's also indicated in some applications where security is sensitive, or
for devices that need to operate in environments where moving parts are
impractical.

Sometimes, a physical (as opposed to virtual, in-kernel-only) RAM
disk is appropriate. That is, you have a piece of hardware you
plug in, which looks and acts like a disk as far as the system is
concerned, but actually stores data in RAM.

While many systems (including vxWorks, as mentioned above) do
include in-main-memory pseudo-disk devices, in a purely theoretical
sense, there is *never* [%] any reason to use one: whatever you
are doing that is going through some sort of "file system" layer,
only to wind up going directly to regular old ordinary memory, is
going through a "unnecessary" manipulation. File systems do a lot
of work in order to deal with structured, and somewhat klunky, disk
drive hardware, where memory exists in the form of "disk blocks" or
"sectors" that can only be rewritten in entire units, cannot be
moved or resized, and is generally very slow to access (about 5 or
6 decimal orders of magnitude slower than RAM). Real RAM does not
share most of these characteristic drawbacks, and going through
a (usually quite heavy) software layer that simply pretends to add
some of them back is just silly.

[% One obvious exception is when the RAM-disk is being used for
testing the file system software. Here, even in theory, the RAM
drive is useful. ]

In practice, the reason for using a RAM disk is that the stubborn
software (written by some stubborn and/or or long-gone programmer)
insists on using the file system interface, and the system provides
no way to short-circuit this. For instance, in C -- to get at
least marginally back on topic -- one might have a program (or
large library) that does all its I/O to a "stdio" stream. Since
ISO C has no way to "open memory as a stream", one is often forced
to supply an actual, on-disk file. To get decent performance, one
may wish to use an "on-RAM-disk file" instead of "on-actual-disk".

Note that it can make sense to use a "RAM file system" rather than
a "RAM disk" in this case. Going through a *real* file system,
you may do a whole lot of extra work trying to avoid talking too
much to a slow (real) disk -- i.e., spend large amounts of CPU time
to avoid waiting for the device. But if the "device" is fake, and
actually is very fast (relative to real disks), this CPU time is
being wasted. It may be better to go through a fake file system
(one which uses RAM), instead of a real file system on a fake disk.

Of course, it would be even better still if C had some sort of
memory-oriented stdio streams -- or something like the 4.4BSD
"funopen", which is even more general. (I am pretty sure that
funopen() did get proposed for C99; but we might consider ourselves
lucky to have gotten even snprintf(). )

Well written and detailed. Thank you Chris. I agree fully with you and
wish to elaborated below a little to make sure I have not missed the
boat.

I have revisited creating a very large array for holding raw data
which is used in an iterative optimization algorithm. The purpose of
the large array is to eliminate the slow hard disk reads and increase
program throughput. On an older machine (Win2000, Visual C++ 6.0, 256
M ram) when I compile with a huge array I get the following warning:

=====================================================================
warning LNK4084: total image size 343744512 exceeds max (268435456);
image may not run
=====================================================================

The help doco on a newer machine (Win2000, Visual C++.NET, 1 G ram)
indicates that the image can be as large as 1 G byte. (This machine is
in the middle of a multi-day run and I do not have a compilation
error/warning available at this time for this system.)

To get around the image size limits and to speed up I/O it seems a ram
disk is one valid approach. A search of ram disk drivers shows that
some can create disks 64+ Gig in size.

I am guessing on my older machine if I were to increase the ram that I
would still be up against the same image size limitation. Thus the ram
disk provides a great improvement for I/O for an iterative type
process and allows you to work around image size limits?

I wish I knew how to load the data into a block of memory and have it
treated as if were the cache for the hard drive. In other words, ready
to use and no manipulation needed. No double handling: caching,
paging, etc.

Chris's above reference to wishing for "funopen" combined with my
image size problems leads me to believe that a ram disk is my only
valid option at this time? Unless I were skilled enough to write my
own funopen! That'll be the day!!

In the future the 64 bit machines may allow much larger image sizes?
Then the data can loaded into arrays as long as memory resources are
available. And only if the data is in nicely proportioned blocks (i.e.
a data structure.) Arrays would not be the optimal solution for loose
form, varying length or delimited type of data. In that case you
really need to be able to assign a block of ram for I/O. I found the
following 64 bit article interesting:

http://msdn.microsoft.com/vstudio/java/compare/benchmark64/default.aspx

Thus for my immediate needs and current inability to write my own
"funopen" ... I must use a ram disk to speed up the I/O on my data
hungry application?

Can anyone recommend their favorite vendor for a ram disk driver?
There seem to be several available but I have yet to find a review on
which is the best and most stable.

Thanks. -- Tom

 

[Kenny McCormack]

....
Nick Keighley makes stupid racist comments on comp.lang.c just to
prove what an idiot he is. This is known as "typical behavior for him";
it is best to make your own choices, as a free rational adult.

 

[Thomas J. Gritzan]

I have revisited creating a very large array for holding raw data
which is used in an iterative optimization algorithm. The purpose of
the large array is to eliminate the slow hard disk reads and increase
program throughput. On an older machine (Win2000, Visual C++ 6.0, 256
M ram) when I compile with a huge array I get the following warning:

=====================================================================
warning LNK4084: total image size 343744512 exceeds max (268435456);
image may not run
=====================================================================

Thats a linker error. What did you do to the linker?

Just don't use a filescope array, try to malloc() a block of memory, or if
malloc() can't handle that large blocks, allocate multiple chunks of
smaller memory blocks.

You really don't need a ramdisk.

 

[pete]

Nick Keighley wrote:

Kenny McCormack makes silly remarks on comp.lang.c just to get a
reaction. This is known as "trolling"; it is best to ignore him.

I agree.

 

[Tom]

Thats a linker error. What did you do to the linker?

Just don't use a filescope array, try to malloc() a block of memory, or if
malloc() can't handle that large blocks, allocate multiple chunks of
smaller memory blocks.

You really don't need a ramdisk.

From Tom to Tom --

Thanks!

My linker is just in the default setting mode. I have never adjusted
it. I write "console" apps.

I need to wait until my current run finishes (another painful 28
hrs)... but as soon as it does I will trying to malloc() my way to
success.

If this works ... I will be thrilled!

 

[Malcolm]

I'd greatly appreciate advice and code snippets on how to create a ram
disk within a C/C++ program.

I also need to be able to determine the free space.

Thanks in advance for any help.

To create ramdisk

unsigned char * ramdisk = malloc(1024 * 64);

if that proves too small

temp = realloc(ramdisk, 1024 * 128);
if(temp)
ramdisk = temp;
else
/* tell user you are very sorry but there is no more memory */

Now we can implement
int ramputc(int ch)
{
if( ch == EOF)
/* do you wnat to handle this ? */
if(len < capacity)
ramdisk[len++] = (unsigned char) ch;
else
return EOF;
return 0;
}

ramfputs, ramfwrite and so on can be built on top of ramputc.

What you cannot do is associate your ramdisk with a C FILE *. Very stupid
restriction, I know. In C++ you can theoretically override the IoStream
classes, but rather you than me.

 

[osmium]

I'd greatly appreciate advice and code snippets on how to create a ram
disk within a C/C++ program.

I also need to be able to determine the free space.

I haven't followed this thread, so this may just repeat what someone has
already said.

In order to create a Ram Disk on a modern system you need the ability to
"nail" some space in the RAM caching mechanism provided by the OS. Lacking
such a capability, the OS is free to roll the contents of what you think is
a RAM address out to disk.

The OS people have already done what they think is a decent job of handling
both internal and external memory via. RAM caching and disk caching.
Anything you do is going to have to fight the mechanisms already in place.
The ability to nail is, IMO, a requisite of an adult OS, which should allow
some real time usage on the system intermixed with the ordinary flow of
lower priority jobs. IOW, the ability might be offered on something like,
say, Windows NT but not on the ordinary line of OSes. I am only familiar
with one disk caching system, and it did not provide an ability to either
nail or tack - where a tack is the short term equivalent of a nail.