Standard Read and Write FILE functions to refer to memory

B

ben beroukhim

I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.

I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.

Ben Beroukhim
 
F

Flash Gordon

On 30 Nov 2004 11:54:01 -0800
I have huge number of legacy code which use standard files functions.
I would like to pass a memory pointer rather than a FILE pointer.

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.

Since the C file functions are file functions they operate on files.
I do not want to touch the legacy code. Does any one know of a library
to do this. I would appreciate the help.

No library can turn files in to anything other than files. Even if such
a thing were possible, it would still be off topic for this group since
we only discuss standard C.

<OT>
You could ask in a group dedicated to your system about RAM disks.
</OT>
 
B

Ben Pfaff

I am trying to use FILEs in the code to refer to memory
buffers. Basically, I want to be able to use all the standard
read and write functions, but I want them to refer to memory
locations, rather than disk files.

There is no standard way to do this. Many implementations of the
C library offer extensions that allow it, however. You should
refer to the reference manual for your C implementation to see if
yours is one of them.
 
K

Keith Thompson

Flash Gordon said:
On 30 Nov 2004 11:54:01 -0800


Since the C file functions are file functions they operate on files.

Sure, but what is a "file"?

There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.

In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.
 
D

dandelion

There's no reason a C implementation couldn't provide a way to refer
to a memory buffer as if it were a file. For example, it could
establish a convention that any file name starting with "mem:" refers
to a memory buffer.

Hmmmm.... Not sure wether that's a good idea...
In a sense, it's just like asking how to use FILEs to refer to files
stored on a hard drive (except that almost all C implementations
support such an interface).

This is entirely implementation-specific, of course. There's no way
to do it in standard C that's going to work on all implementations.

Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().

If they are called with a reference to the current location in the
file-buffer and a count of bytesto write (which isn't hard to manage), the
usual stdio suspects can be used to write data anywhere at an application
level.

Unbuffered FILE's call the same routines for each byte read or written, thus
behaving as a one byte buffer.

On the downside: ungetc() would pose a problem.

dandelion

Disclaimer: This idea has been nicked off C++. All the merit belongs there.
 
C

CBFalconer

dandelion said:
.... snip ...

Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

Both could easily (and portably) be set using fcntl().

No it can't. One possible reason is that fcntl() is not standard.
 
D

dandelion

CBFalconer said:
No it can't. One possible reason is that fcntl() is not standard.

Right. The rest of the idea isn't standard, either (but that's on purpose).
I got Posix confused with C standards again, it seems. Still, the idea of a
couple of callbacks for the purpose mentioned seems a good idea.
 
J

Jonathan Bartlett

I am trying to use FILEs in the code to refer to memory buffers.
Basically, I want to be able to use all the standard read and write
functions, but I want them to refer to memory locations, rather than
disk
files.

Just an idea -- use a temporary file and then mmap it in.

Jon
 
C

Chris Torek

Defining a couple of hooks would be sufficient:
- file_overflow to provide the 'write' functionality and
- file_underflow to provide 'read' functionality.

I put this sort of thing, too, into the 4.3BSD stdio. In addition
to snprintf() (which did get into C99), I have "funopen":

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" (const-qualified in funopen but not
in the underlying functions), and the read, write, seek, and close
function parameters are pointers to functions that implement reading,
writing, seeking, and closing. Each function is optional; passing
NULL means "nothing to do for this". You cannot pass NULL for both
read and write functions -- using NULL for one of them makes it a
read-only or write-only stream as appropriate; using NULL for both
would make it a close-only stream. :) Giving NULL as the seek
function makes it an unseekable stream (fseek() returns errors).

The cookie is simply passed back to your functions, giving you a
place to store data you want to associate with that particular stream,
e.g., the memory area for a "memory-stream", or window data
structure for a window-system output-message window, or compression
data for a zip/unzip stream, etc. Note that a compression stream
would typically sit atop a regular-file stream:

struct compression_data {
FILE *lower_file;
... rest of compression data ...
};

int compressor_write(void *cookie, const char *buf, int len) {
struct compression_data *cd = cookie;
... compress the data from buf[] using whatever algorithm ...
... put each character out via putc(c, cd->lower_file),
or fwrite(ptr, size, n, cd->lower_file) ...
return n_written_successfully;
}

but of course the "lower file" could be anything stdio handles,
including a memory stream, or an encryptor stream, or a binary-to-base64
mime-encoder stream, etc. It might well be reasonable to stack a
compressor above an encryptor that sits above a mime-encoder (and,
conversely, a mime-decoder below a decryptor below a decompressor).

For whatever reason, these "stackable" streams did not get into
C99.
 
C

CBFalconer

Keith said:
dandelion said:
Hmmmm.... Not sure wether that's a good idea...
[...]

I wrote the above; you snipped the attribution.

Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.

Corollary: Any ugly looking article is not worth reading.
 
D

dandelion

CBFalconer said:
Keith said:
dandelion said:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea...
[...]

I wrote the above; you snipped the attribution.

Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.

Here's another: People who write vindictive 'laws' are anal retentive.
Please look up the definition of the word 'Theory' before you postulate any,
especially accusing others of whowing lack of attention to detail.
Corollary: Any ugly looking article is not worth reading.

Unfortunately You don't know that before you read it. Anything of substance
to add or will you stick you your usual whining today?
 
D

dandelion

CBFalconer said:
Keith said:
dandelion said:
There's no reason a C implementation couldn't provide a way to
refer to a memory buffer as if it were a file. For example, it
could establish a convention that any file name starting with
"mem:" refers to a memory buffer.

Hmmmm.... Not sure wether that's a good idea...
[...]

I wrote the above; you snipped the attribution.

Let me offer a theory: People who are incapable of properly
snipping and attributing quotes are very unlikely to be able to
cope with the vagaries of the C language. They show an appalling
lack of attention to detail.

Here's another: People who write vindictive 'laws' are anal retentive.
 
S

Steve Summit

Those are the sort of things you want if you're trying
specifically to read/write from/to strings. The more usual
hooks are general-purpose write and read functions (though doing
it that way, it's true, does not end up easily supporting the
additional goal of writing to a malloc'ed region of memory that
grows as needed).

Chris Torek replied:
I put this sort of thing, too, into the 4.3BSD stdio...

FILE *funopen(cookie, readfn, writefn, seekfn, closefn)

where "cookie" is a "void *" and the... function parameters are
pointers to functions that implement reading, writing, seeking,
and closing.

It turns out that the GNU folks have implemented this, also,
in at least one of the versions of glibc. The recipe is slightly
different:

cookie_io_functions_t funcs = {readfn, writefn, seekfn, closefn};
FILE *fp = fopencookie(cookie, mode, funcs);

where mode is "r" or "w", as usual.

Since this is an extension, you have to define the macro
_GNU_SOURCE to enable it. Once you've done so, you also have
access to

FILE *fmemopen(void *str, size_t len, const char *mode)

which gives just the functionality the original poster was asking
for (i.e. without having to define any of your own auxiliary read
or write functions), and also

FILE *open_memstream(char **strp, size_t *lenp)

which writes to the malloc'ed region pointed to by *strp and
grows it as needed (updating the size *lenp as it does so).
I don't know if any of this is documented anywhere; I stumbled
across it only recently while looking for something else in GNU's
copy of <stdio.h>.

Anyway, if you're using Linux or some other glibc-using system,
this functionality may well be available to you already. (But
before the topicality police come down on me, I have to reiterate
that these extensions are not standard and that code which makes
use of them is therefore not portable.)
For whatever reason, these "stackable" streams did not get into C99.

And a pity it is, because it's insanely useful functionality,
and not difficult or expensive to implement. It seems like
practically every large program I've ever written has needed to
do this sort of thing (i.e. read or write interchangeably to a
file or a string) at some point; half the time I end up writing
my own little ad-hoc wrapper on top of stdio to let me do this,
which is doubly inefficient.

Steve Summit
(e-mail address removed)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top