how to save a file to memory rather to disk?

Discussion in 'C++' started by zl2k, Mar 28, 2011.

  1. zl2k

    zl2k Guest

    hi, there,

    Is there any way that I can save a file (a few k) in to the memory
    than on disk (and then delete it)? Thanks for help.

    zl2k
    zl2k, Mar 28, 2011
    #1
    1. Advertising

  2. On 3/28/2011 11:04 AM, zl2k wrote:
    > hi, there,
    >
    > Is there any way that I can save a file (a few k) in to the memory
    > than on disk (and then delete it)? Thanks for help.


    What do you mean by "save a file"? You can have an ostringstream and
    write to it. Read up on string-based streams.

    V
    --
    I do not respond to top-posted replies, please don't ask
    Victor Bazarov, Mar 28, 2011
    #2
    1. Advertising

  3. zl2k

    zl2k Guest

    On Mar 28, 10:21 am, cg_chas <> wrote:
    > On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <> wrote:
    > >hi, there,

    >
    > >Is there any way that I can save a file (a few k) in to the memory
    > >than on disk (and then delete it)? Thanks for help.

    >
    > >zl2k

    >
    > I am not 100% sure what you are asking here since "saving a file in to the
    > memory" seems to be a bit of an odd way to say load a file into memory.  In
    > which case the anwer is yes.
    >
    > If you know the size of the data you are reading, you can construct a
    > vector<unsigned char>(size) and call fstream::read passing it a pointer to the
    > first element of the vector. i.e. loads it into memory.  When you're done, you
    > can fstream::write it to a file on disk., let the vector go out of scope or
    > delete it depending on how it was allocated initially.


    Please allow me to clarify the question. In my code it generates a jpg
    image, and another api will consume the jpg. I can write the jpg to
    disk as "abc.jpg" and pass that to the consumer program, but that may
    cost IO time. What I think is if there is a way I can pass it to the
    consumer program without the disk IO cost. Thanks.
    zl2k, Mar 28, 2011
    #3
  4. zl2k

    Öö Tiib Guest

    On Mar 28, 7:38 pm, zl2k <> wrote:
    >
    > Please allow me to clarify the question. In my code it generates a jpg
    > image, and another api will consume the jpg. I can write the jpg to
    > disk as "abc.jpg" and pass that to the consumer program, but that may
    > cost IO time. What I think is if there is a way I can pass it to the
    > consumer program without the disk IO cost.


    Then you take and read up on named pipes

    http://en.wikipedia.org/wiki/Named_pipe
    Öö Tiib, Mar 28, 2011
    #4
  5. zl2k

    Jorgen Grahn Guest

    On Mon, 2011-03-28, zl2k wrote:
    > On Mar 28, 10:21 am, cg_chas <> wrote:
    >> On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <> wrote:
    >> >hi, there,

    >>
    >> >Is there any way that I can save a file (a few k) in to the memory
    >> >than on disk (and then delete it)? Thanks for help.

    >>
    >> >zl2k

    >>
    >> I am not 100% sure what you are asking here since "saving a file in to the
    >> memory" seems to be a bit of an odd way to say load a file into memory.  In
    >> which case the anwer is yes.

    ....

    > Please allow me to clarify the question. In my code it generates a jpg
    > image, and another api will consume the jpg. I can write the jpg to
    > disk as "abc.jpg" and pass that to the consumer program, but that may
    > cost IO time. What I think is if there is a way I can pass it to the
    > consumer program without the disk IO cost. Thanks.


    That's still an unclear description which partly contradicts your
    first one. I guess you mean this:

    "My program (A) can generate an image, but it needs to pass it in
    jpeg format to this other program (B), using some unspecified
    interface of B. I can do this by storing the image to the file
    system (B handles that case), but I want to avoid that."

    I'd say it depends entirely on that other program, and your OS.
    If you're lucky you're on Unix and B can read the image from standard
    input. If you're unlucky B requires a seekable file.

    I don't like doing IPC via the file system either, but it's not so
    much an issue of I/O cost. I don't like it because
    - it kills parallelism: B cannot start its work until A has written
    everything
    - the file system size sets a limit; I cannot process 10GB pictures
    if I don't have 10GB free disk space
    - I risk leaving garbage lying around in the file system
    if either process crashes or is killed by the user.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
    Jorgen Grahn, Mar 28, 2011
    #5
  6. zl2k

    Sjouke Burry Guest

    zl2k wrote:
    > hi, there,
    >
    > Is there any way that I can save a file (a few k) in to the memory
    > than on disk (and then delete it)? Thanks for help.
    >
    > zl2k

    Open a ramdisk at startup.
    Then treat it as any other drive.
    Sjouke Burry, Mar 28, 2011
    #6
  7. zl2k

    James Kanze Guest

    On Mar 28, 5:38 pm, zl2k <> wrote:
    > On Mar 28, 10:21 am, cg_chas <> wrote:
    > > On Mon, 28 Mar 2011 08:04:48 -0700 (PDT), zl2k <> wrote:


    > > >Is there any way that I can save a file (a few k) in to the
    > > >memory than on disk (and then delete it)? Thanks for help.


    > > I am not 100% sure what you are asking here since "saving a
    > > file in to the memory" seems to be a bit of an odd way to
    > > say load a file into memory. In which case the anwer is
    > > yes.


    > > If you know the size of the data you are reading, you can
    > > construct a vector<unsigned char>(size) and call
    > > fstream::read passing it a pointer to the first element of
    > > the vector. i.e. loads it into memory. When you're done,
    > > you can fstream::write it to a file on disk., let the vector
    > > go out of scope or delete it depending on how it was
    > > allocated initially.


    > Please allow me to clarify the question. In my code it generates a jpg
    > image, and another api will consume the jpg. I can write the jpg to
    > disk as "abc.jpg" and pass that to the consumer program, but that may
    > cost IO time.


    Or it may not. On most systems, if the file isn't too big, and
    is deleted quickly enough after it is written, it may never hit
    the disk. Or you can use shared memory---but if the data hangs
    around enough in shared memory, it may end up on disk. In
    general, the OS makes the decision about such things, not you
    (although you can generally force it when necessary, e.g. for
    reasons of transactional integrity).

    --
    James Kanze
    James Kanze, Mar 28, 2011
    #7
  8. zl2k

    puppi Guest

    You could try using shared memory, by looking into the functions
    mmap() and shmget(). However, note that it's quite a low level
    approach. Also, they are not Standard C++ functions, they are POSIX's.
    If you have specific OSs in mind, such as Linux, then it's an
    alternative. Just don't expect the portability to be the same as using
    the filesystem.
    puppi, Mar 29, 2011
    #8
  9. On Mar 28, 3:36 pm, James Kanze <> wrote:
    > On Mar 28, 5:38 pm, zl2k <> wrote:
    > > Please allow me to clarify the question. In my code it generates a jpg
    > > image, and another api will consume the jpg. I can write the jpg to
    > > disk as "abc.jpg" and pass that to the consumer program, but that may
    > > cost IO time.

    >
    > Or it may not.  On most systems, if the file isn't too big, and
    > is deleted quickly enough after it is written, it may never hit
    > the disk.  Or you can use shared memory---but if the data hangs
    > around enough in shared memory, it may end up on disk.  In
    > general, the OS makes the decision about such things, not you
    > (although you can generally force it when necessary, e.g. for
    > reasons of transactional integrity).


    Allow me to take disagreement with your points in parentheses. From
    what little reading I've done, ensuring an actual flush to non-
    volatile storage on POSIX OSs and win32 is not easy. Across all
    platforms, you do not have a real-life guarantee that sync, fsync, and
    fdatasync force the data to non-volatile storage. The last time I
    researched this problem, I learned that there are no easy answers. It
    depends on the exact OS, its current configuration, its available non-
    standard APIs, and whether or not your exact hard disk model and hard
    disk driver want to play nice.

    In short, I suggest that people do their research before attempting to
    write some software with guarantees in the face of failures from power
    loss or other catastrophic program or OS failures.
    Joshua Maurice, Mar 29, 2011
    #9
  10. zl2k

    James Kanze Guest

    On Mar 29, 8:35 pm, Joshua Maurice <> wrote:
    > On Mar 28, 3:36 pm, James Kanze <> wrote:


    > > On Mar 28, 5:38 pm, zl2k <> wrote:
    > > > Please allow me to clarify the question. In my code it generates a jpg
    > > > image, and another api will consume the jpg. I can write the jpg to
    > > > disk as "abc.jpg" and pass that to the consumer program, but that may
    > > > cost IO time.


    > > Or it may not. On most systems, if the file isn't too big, and
    > > is deleted quickly enough after it is written, it may never hit
    > > the disk. Or you can use shared memory---but if the data hangs
    > > around enough in shared memory, it may end up on disk. In
    > > general, the OS makes the decision about such things, not you
    > > (although you can generally force it when necessary, e.g. for
    > > reasons of transactional integrity).


    > Allow me to take disagreement with your points in parentheses. From
    > what little reading I've done, ensuring an actual flush to non-
    > volatile storage on POSIX OSs and win32 is not easy. Across all
    > platforms, you do not have a real-life guarantee that sync, fsync, and
    > fdatasync force the data to non-volatile storage. The last time I
    > researched this problem, I learned that there are no easy answers. It
    > depends on the exact OS, its current configuration, its available non-
    > standard APIs, and whether or not your exact hard disk model and hard
    > disk driver want to play nice.


    Well, it's not portable, that's for sure. And my actual
    experience in this domain is limited to Solaris Sparc. But as
    long as the disk was local (no NFS involved), it seemed to work.
    Or perhaps we were just lucky.

    Posix does make specific requirements, as well; presumably, a
    system where it didn't work wouldn't be Posix compliant. But of
    course, there are always bugs, and if you're remote mounting
    drives, the system is dependent on what the protocol provides,
    and how well the remote host complies, regardless of what Posix
    says, or even how well it has been written itself.

    > In short, I suggest that people do their research before attempting to
    > write some software with guarantees in the face of failures from power
    > loss or other catastrophic program or OS failures.


    Definitly. It's important to understand the specific platform
    you're programming to, and what guarantees it actually gives.
    And to maintain a good deal of scepticism; I'm not aware of any
    specific issues with NFS, for example, but as soon as two or
    more entities are involved, I become very mistrustful.

    --
    James Kanze
    James Kanze, Apr 3, 2011
    #10
  11. zl2k

    Cholo Lennon Guest

    On 29/03/2011 15:51, puppi wrote:
    > You could try using shared memory, by looking into the functions
    > mmap() and shmget(). However, note that it's quite a low level
    > approach. Also, they are not Standard C++ functions, they are POSIX's.
    > If you have specific OSs in mind, such as Linux, then it's an
    > alternative. Just don't expect the portability to be the same as using
    > the filesystem.


    The OP could try using Boost.Interprocess library if he/she wants to use
    a portable shared memory approach.

    Regards

    --
    Cholo Lennon
    Bs.As.
    ARG
    Cholo Lennon, Apr 4, 2011
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    484
    perry
    May 13, 2004
  2. tiewknvc9
    Replies:
    12
    Views:
    1,262
    Chris Uppal
    Feb 19, 2007
  3. CMOS
    Replies:
    15
    Views:
    468
    James Kanze
    May 17, 2007
  4. nick
    Replies:
    58
    Views:
    1,900
    Bart van Ingen Schenau
    Mar 16, 2009
  5. Gelonida N
    Replies:
    5
    Views:
    296
    Tim Chase
    Oct 29, 2011
Loading...

Share This Page