Re: File read function from memory AND a file?

Discussion in 'C++' started by James Kanze, Feb 10, 2011.

  1. James Kanze

    James Kanze Guest

    On Feb 10, 6:14 pm, Bint <> wrote:

    > I have a function which reads a file using std::ifstream infile, and a
    > bunch of functions like this:


    > infile.read((char *)&f,sizeof(float))


    > I would like to be able to use the same function to read files
    > that have been downloaded -- they only reside in memory.


    > Is there any way to use the same function to read from both
    > memory and a file?


    Of course.

    > I read about stringstreams, but so far I haven't figured out
    > if I can pass a stringstream pointer to a function expecting
    > an ifstream. Can I?


    A function should almost never take a pointer or a reference to
    an std::ifstream. It should take a pointer or a reference to an
    std::istream, so that it can be passed an ifstream, an
    istringstream, or any other type of istream you might have.

    --
    James Kanze
     
    James Kanze, Feb 10, 2011
    #1
    1. Advertising

  2. James Kanze

    Jorgen Grahn Guest

    On Mon, 2011-02-14, Bint wrote:
    ....
    >>
    >> OK -- I changed my function to take an istream, and reading from a file
    >> still works. Reading from memory looks like it will work, except that I
    >> was previously checking for errors by whether infile.read() returned 0 or
    >> not. I had read that ifstream at least returned 0 on error. But it seems
    >> like istream does not.
    >>
    >> I can change my function to ignore the return value, but then what is the
    >> proper way to error check this type of function?
    >>
    >> Thanks
    >> B

    >
    > I found out how to check the error status using infile.good(). The problem
    > is that my memory buffer isn't good! At least, after the first few reads,
    > the infile.good() function is returning 0, and a check on the rdstate
    > reveals that the failbit and the eofbit are set.


    You should find better reference material. f.good() means "next
    operation might succeed" and you missed f.bad() which is the one
    that flags true I/O errors.

    > But this happens consistently after I've read 7 bytes from the memory
    > buffer. I read three individual bytes, for which good() is ok. Then I read
    > an int, and it gets the value right but after that good() returns the
    > aforementioned errors.
    >
    > I have checked and the buffer is 1697 bytes long, and the values all look
    > right. It just seems that istringstream doesn't know how long the buffer
    > is.
    >
    > I guess since it is a "string" stream, is it just assuming the first 0 it
    > hits is the end of the file?! If that's the case, how can I use this
    > function to read binary file data? There are plenty of zeros in there.


    I'm pretty sure you have some kind of bug on your side. std::string
    doesn't treat '\0' specially, except when converting to C strings
    which do.

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Feb 14, 2011
    #2
    1. Advertising

  3. James Kanze

    Nobody Guest

    On Mon, 14 Feb 2011 10:14:39 -0600, Bint wrote:

    > I have checked and the buffer is 1697 bytes long, and the values all look
    > right. It just seems that istringstream doesn't know how long the buffer
    > is.
    >
    > I guess since it is a "string" stream, is it just assuming the first 0 it
    > hits is the end of the file?! If that's the case, how can I use this
    > function to read binary file data? There are plenty of zeros in there.


    C++ strings (i.e. std::string) aren't NUL-terminated; they have an
    explicit size.

    std::string has two constructors which create a std::string from a char*:

    string ( const char * s, size_t n );
    string ( const char * s );

    The first one should be used if you have an array of known length (which
    may or may not contain NUL bytes). The second one should be used for a
    NUL-terminated (C-style) string, and will be used for implicit conversions
    (i.e. if you pass a char* where a std::string is expected, e.g. the
    istringstream constructor).

    Also, note that the simplistic approach of:

    istringstream iss(string(ptr, len));

    will copy the data (twice). You can avoid this by setting the buffer of
    the underlying streambuf object, e.g.:

    istringstream iss;
    iss.rdbuf()->pubsetbuf(ptr, len);
     
    Nobody, Feb 14, 2011
    #3
  4. James Kanze

    James Kanze Guest

    On Feb 14, 4:14 pm, Bint <> wrote:
    > > On 2/10/11 12:56 PM, in article
    > > , "James
    > > Kanze" <> wrote:


    > > OK -- I changed my function to take an istream, and reading from a file
    > > still works. Reading from memory looks like it will work, except that I
    > > was previously checking for errors by whether infile.read() returned 0 or
    > > not. I had read that ifstream at least returned 0 on error. But it seems
    > > like istream does not.


    There is no ifstream::read nor istringstream::read. You're
    calling exactly the same function in both cases: istream::read.
    If the results are different, your data is different.

    Also, read doesn't return an int, but rather something that will
    convert to bool (a pointer, in fact, but this is irrelevant,
    since regardless of its value, you're not allowed to dereference
    it).

    > > I can change my function to ignore the return value, but then what is the
    > > proper way to error check this type of function?


    > I found out how to check the error status using infile.good().


    That is *not* the right way to check error status. The only right
    way is to use istream::fail(). Or the implicite conversion to
    pointer, or the ! operator, both of which use istream::fail().

    > The problem
    > is that my memory buffer isn't good! At least, after the first few reads,
    > the infile.good() function is returning 0, and a check on the rdstate
    > reveals that the failbit and the eofbit are set.


    If failbit is set, then you're input has failed.

    > But this happens consistently after I've read 7 bytes from the memory
    > buffer. I read three individual bytes, for which good() is ok. Then I read
    > an int, and it gets the value right but after that good() returns the
    > aforementioned errors.


    > I have checked and the buffer is 1697 bytes long, and the values all look
    > right. It just seems that istringstream doesn't know how long the buffer
    > is.


    How do you initialize it?

    > I guess since it is a "string" stream, is it just assuming the first 0 it
    > hits is the end of the file?!


    Since it is a string stream, it doesn't take a memory buffer,
    but rather a string. Which constructor of std::string are you
    using?

    You might also look into istrstream. Officially, this is
    deprecated, but practically, it can be very useful, and isn't
    likely to disappear. Or you can write your own memory based
    streambuf (about 10 lines of code), and use it. (For 1697
    bytes, it's probably not worth the bother, but when the memory
    buffer measures in mega-bytes, not copying becomes essential.)

    > If that's the case, how can I use this
    > function to read binary file data? There are plenty of zeros in there.


    An std::string can contain zeros. Just construct the string
    with a pointer to the start and the length, or a pointer to the
    start and a pointer to one past the end.

    --
    James Kanze
     
    James Kanze, Feb 15, 2011
    #4
  5. James Kanze

    James Kanze Guest

    On Feb 14, 6:51 pm, Jorgen Grahn <> wrote:
    > On Mon, 2011-02-14, Bint wrote:
    > >> OK -- I changed my function to take an istream, and reading
    > >> from a file still works. Reading from memory looks like it
    > >> will work, except that I was previously checking for errors
    > >> by whether infile.read() returned 0 or not. I had read
    > >> that ifstream at least returned 0 on error. But it seems
    > >> like istream does not.


    > >> I can change my function to ignore the return value, but
    > >> then what is the proper way to error check this type of
    > >> function?


    > > I found out how to check the error status using
    > > infile.good(). The problem is that my memory buffer isn't
    > > good! At least, after the first few reads, the
    > > infile.good() function is returning 0, and a check on the
    > > rdstate reveals that the failbit and the eofbit are set.


    > You should find better reference material. f.good() means "next
    > operation might succeed" and you missed f.bad() which is the one
    > that flags true I/O errors.


    But you won't get a true I/O error when reading a stream.
    f.fail() is the one he's interested in: this will return true in
    all cases of failure (including end of file).

    --
    James Kanze
     
    James Kanze, Feb 15, 2011
    #5
  6. James Kanze

    James Kanze Guest

    On Feb 14, 6:55 pm, Nobody <> wrote:
    > On Mon, 14 Feb 2011 10:14:39 -0600, Bint wrote:
    > > I have checked and the buffer is 1697 bytes long, and the values all look
    > > right. It just seems that istringstream doesn't know how long the buffer
    > > is.


    > > I guess since it is a "string" stream, is it just assuming the first 0 it
    > > hits is the end of the file?! If that's the case, how can I use this
    > > function to read binary file data? There are plenty of zeros in there.


    > C++ strings (i.e. std::string) aren't NUL-terminated; they have an
    > explicit size.


    > std::string has two constructors which create a std::string from a char*:


    > string ( const char * s, size_t n );
    > string ( const char * s );


    Don't forget
    template< typename Iterator > string( Iterator begin, Iterator
    end );
    , with char const* or char* as Iterator.

    > The first one should be used if you have an array of known length (which
    > may or may not contain NUL bytes). The second one should be used for a
    > NUL-terminated (C-style) string, and will be used for implicit conversions
    > (i.e. if you pass a char* where a std::string is expected, e.g. the
    > istringstream constructor).


    > Also, note that the simplistic approach of:


    > istringstream iss(string(ptr, len));


    > will copy the data (twice). You can avoid this by setting the buffer of
    > the underlying streambuf object, e.g.:


    > istringstream iss;
    > iss.rdbuf()->pubsetbuf(ptr, len);


    It shouldn't. A streambuf isn't required to accept calls to
    pubsetbuf, and I suspect that most implementations of
    istringstream will ignore them, or if it uses them, will copy
    the actual string into the given buffer. Setting the buffer
    doesn't change the current state of any of the buffer pointers;
    it only tells the implementation that it should use this memory
    for the buffer, rather than memory it has obtained elsewhere.

    If avoiding the copies is important, he can always use
    istrstream, or his own memory based streambuf.

    --
    James Kanze
     
    James Kanze, Feb 15, 2011
    #6
  7. James Kanze

    Jorgen Grahn Guest

    On Tue, 2011-02-15, James Kanze wrote:
    > On Feb 14, 6:51 pm, Jorgen Grahn <> wrote:
    >> On Mon, 2011-02-14, Bint wrote:
    >> >> OK -- I changed my function to take an istream, and reading
    >> >> from a file still works. Reading from memory looks like it
    >> >> will work, except that I was previously checking for errors
    >> >> by whether infile.read() returned 0 or not. I had read
    >> >> that ifstream at least returned 0 on error. But it seems
    >> >> like istream does not.

    >
    >> >> I can change my function to ignore the return value, but
    >> >> then what is the proper way to error check this type of
    >> >> function?

    >
    >> > I found out how to check the error status using
    >> > infile.good(). The problem is that my memory buffer isn't
    >> > good! At least, after the first few reads, the
    >> > infile.good() function is returning 0, and a check on the
    >> > rdstate reveals that the failbit and the eofbit are set.

    >
    >> You should find better reference material. f.good() means "next
    >> operation might succeed" and you missed f.bad() which is the one
    >> that flags true I/O errors.

    >
    > But you won't get a true I/O error when reading a stream.
    > f.fail() is the one he's interested in: this will return true in
    > all cases of failure (including end of file).


    That's true I guess, but if he wants code which works with istreams,
    he still needs to check for and handle actual I/O errors, in case it
    happens to be an ifstream (or some other stream which is based on real
    I/O).

    I hate programs which treat I/O errors as end-of-file. (In practice,
    since disk read errors are so rare, programs which treat
    read-protected or missing files as empty files.)

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
     
    Jorgen Grahn, Feb 15, 2011
    #7
  8. James Kanze

    James Kanze Guest

    On Feb 15, 3:00 pm, Jorgen Grahn <> wrote:
    > On Tue, 2011-02-15, James Kanze wrote:
    > > On Feb 14, 6:51 pm, Jorgen Grahn <> wrote:
    > >> On Mon, 2011-02-14, Bint wrote:
    > >> >> OK -- I changed my function to take an istream, and reading
    > >> >> from a file still works. Reading from memory looks like it
    > >> >> will work, except that I was previously checking for errors
    > >> >> by whether infile.read() returned 0 or not. I had read
    > >> >> that ifstream at least returned 0 on error. But it seems
    > >> >> like istream does not.


    > >> >> I can change my function to ignore the return value, but
    > >> >> then what is the proper way to error check this type of
    > >> >> function?


    > >> > I found out how to check the error status using
    > >> > infile.good(). The problem is that my memory buffer isn't
    > >> > good! At least, after the first few reads, the
    > >> > infile.good() function is returning 0, and a check on the
    > >> > rdstate reveals that the failbit and the eofbit are set.


    > >> You should find better reference material. f.good() means "next
    > >> operation might succeed" and you missed f.bad() which is the one
    > >> that flags true I/O errors.


    > > But you won't get a true I/O error when reading a stream.
    > > f.fail() is the one he's interested in: this will return true in
    > > all cases of failure (including end of file).


    > That's true I guess, but if he wants code which works with istreams,
    > he still needs to check for and handle actual I/O errors, in case it
    > happens to be an ifstream (or some other stream which is based on real
    > I/O).


    Yes, but only after f.fail() returns true. (f.fail() returns (badbit
    |
    failbit) != 0.)

    > I hate programs which treat I/O errors as end-of-file. (In practice,
    > since disk read errors are so rare, programs which treat
    > read-protected or missing files as empty files.)


    Read-protected or missing should cause an error on open; most programs
    I've seen do test immediately after the open, and output an error
    message along the lines of "cannot open xyz.txt".

    Distinguishing between other errors and end of file is essential,
    however, since not doing so could result in data not being processed,
    with no indication of error. In practice, I think a lot of
    implementations treat a disk error as end of file within iostream (or
    am
    I out of date, and this has improved); if the implementation doesn't
    provide the information correctly, you can't act on it. But you also
    want to know if you stopped because of a format error.

    --
    James Kanze
     
    James Kanze, Feb 15, 2011
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page