how to save a file to memory rather to disk?

[zl2k]

hi, there,

Is there any way that I can save a file (a few k) in to the memory
than on disk (and then delete it)? Thanks for help.

zl2k

 

[Victor Bazarov]

hi, there,

Is there any way that I can save a file (a few k) in to the memory
than on disk (and then delete it)? Thanks for help.

What do you mean by "save a file"? You can have an ostringstream and
write to it. Read up on string-based streams.

V

 

[zl2k]

I am not 100% sure what you are asking here since "saving a file in to the
memory" seems to be a bit of an odd way to say load a file into memory.  In
which case the anwer is yes.

If you know the size of the data you are reading, you can construct a
vector<unsigned char>(size) and call fstream::read passing it a pointer to the
first element of the vector. i.e. loads it into memory.  When you're done, you
can fstream::write it to a file on disk., let the vector go out of scope or
delete it depending on how it was allocated initially.

Please allow me to clarify the question. In my code it generates a jpg
image, and another api will consume the jpg. I can write the jpg to
disk as "abc.jpg" and pass that to the consumer program, but that may
cost IO time. What I think is if there is a way I can pass it to the
consumer program without the disk IO cost. Thanks.

 

[Öö Tiib]

Please allow me to clarify the question. In my code it generates a jpg
image, and another api will consume the jpg. I can write the jpg to
disk as "abc.jpg" and pass that to the consumer program, but that may
cost IO time. What I think is if there is a way I can pass it to the
consumer program without the disk IO cost.

Then you take and read up on named pipes

http://en.wikipedia.org/wiki/Named_pipe

 

[Jorgen Grahn]

....

Please allow me to clarify the question. In my code it generates a jpg
image, and another api will consume the jpg. I can write the jpg to
disk as "abc.jpg" and pass that to the consumer program, but that may
cost IO time. What I think is if there is a way I can pass it to the
consumer program without the disk IO cost. Thanks.

That's still an unclear description which partly contradicts your
first one. I guess you mean this:

"My program (A) can generate an image, but it needs to pass it in
jpeg format to this other program (B), using some unspecified
interface of B. I can do this by storing the image to the file
system (B handles that case), but I want to avoid that."

I'd say it depends entirely on that other program, and your OS.
If you're lucky you're on Unix and B can read the image from standard
input. If you're unlucky B requires a seekable file.

I don't like doing IPC via the file system either, but it's not so
much an issue of I/O cost. I don't like it because
- it kills parallelism: B cannot start its work until A has written
everything
- the file system size sets a limit; I cannot process 10GB pictures
if I don't have 10GB free disk space
- I risk leaving garbage lying around in the file system
if either process crashes or is killed by the user.

/Jorgen

 

[Sjouke Burry]

hi, there,

Is there any way that I can save a file (a few k) in to the memory
than on disk (and then delete it)? Thanks for help.

zl2k

Open a ramdisk at startup.
Then treat it as any other drive.

 

[James Kanze]

Please allow me to clarify the question. In my code it generates a jpg
image, and another api will consume the jpg. I can write the jpg to
disk as "abc.jpg" and pass that to the consumer program, but that may
cost IO time.

Or it may not. On most systems, if the file isn't too big, and
is deleted quickly enough after it is written, it may never hit
the disk. Or you can use shared memory---but if the data hangs
around enough in shared memory, it may end up on disk. In
general, the OS makes the decision about such things, not you
(although you can generally force it when necessary, e.g. for
reasons of transactional integrity).

 

[puppi]

You could try using shared memory, by looking into the functions
mmap() and shmget(). However, note that it's quite a low level
approach. Also, they are not Standard C++ functions, they are POSIX's.
If you have specific OSs in mind, such as Linux, then it's an
alternative. Just don't expect the portability to be the same as using
the filesystem.

 

[Joshua Maurice]

Or it may not.  On most systems, if the file isn't too big, and
is deleted quickly enough after it is written, it may never hit
the disk.  Or you can use shared memory---but if the data hangs
around enough in shared memory, it may end up on disk.  In
general, the OS makes the decision about such things, not you
(although you can generally force it when necessary, e.g. for
reasons of transactional integrity).

Allow me to take disagreement with your points in parentheses. From
what little reading I've done, ensuring an actual flush to non-
volatile storage on POSIX OSs and win32 is not easy. Across all
platforms, you do not have a real-life guarantee that sync, fsync, and
fdatasync force the data to non-volatile storage. The last time I
researched this problem, I learned that there are no easy answers. It
depends on the exact OS, its current configuration, its available non-
standard APIs, and whether or not your exact hard disk model and hard
disk driver want to play nice.

In short, I suggest that people do their research before attempting to
write some software with guarantees in the face of failures from power
loss or other catastrophic program or OS failures.

 

[James Kanze]

Allow me to take disagreement with your points in parentheses. From
what little reading I've done, ensuring an actual flush to non-
volatile storage on POSIX OSs and win32 is not easy. Across all
platforms, you do not have a real-life guarantee that sync, fsync, and
fdatasync force the data to non-volatile storage. The last time I
researched this problem, I learned that there are no easy answers. It
depends on the exact OS, its current configuration, its available non-
standard APIs, and whether or not your exact hard disk model and hard
disk driver want to play nice.

Well, it's not portable, that's for sure. And my actual
experience in this domain is limited to Solaris Sparc. But as
long as the disk was local (no NFS involved), it seemed to work.
Or perhaps we were just lucky.

Posix does make specific requirements, as well; presumably, a
system where it didn't work wouldn't be Posix compliant. But of
course, there are always bugs, and if you're remote mounting
drives, the system is dependent on what the protocol provides,
and how well the remote host complies, regardless of what Posix
says, or even how well it has been written itself.

In short, I suggest that people do their research before attempting to
write some software with guarantees in the face of failures from power
loss or other catastrophic program or OS failures.

Definitly. It's important to understand the specific platform
you're programming to, and what guarantees it actually gives.
And to maintain a good deal of scepticism; I'm not aware of any
specific issues with NFS, for example, but as soon as two or
more entities are involved, I become very mistrustful.

 

[Cholo Lennon]

You could try using shared memory, by looking into the functions
mmap() and shmget(). However, note that it's quite a low level
approach. Also, they are not Standard C++ functions, they are POSIX's.
If you have specific OSs in mind, such as Linux, then it's an
alternative. Just don't expect the portability to be the same as using
the filesystem.

The OP could try using Boost.Interprocess library if he/she wants to use
a portable shared memory approach.

Regards