No buffered I/O in the Standard Library?

Discussion in 'C++' started by Steven T. Hatton, Jun 24, 2005.

  1. I haven't looked very closely, but from what I'm seeing, it looks like there
    are no buffered I/O streams in the Standard Library. There are stream
    buffers, but not buffered streams. I don't have an excellent definition of
    what a buffered stream is right off the top of my head, but it's something
    like a cache that can hold data from the source, or for the destination
    that can't be immediately processed. Say, for example, if you are
    streaming to a network socket that sends data in larger chunks than your
    process produces per output operation. You don't want to send an almost
    empty packet every time you write to the stream, then sit around an wait
    till the output stream becomes available again when the network driver
    finishes its transmission. So you write to a buffer from which the network
    driver reads.

    Does the standard library not provide a generic buffered stream?

    --
    If our hypothesis is about anything and not about some one or more
    particular things, then our deductions constitute mathematics. Thus
    mathematics may be defined as the subject in which we never know what we
    are talking about, nor whether what we are saying is true.-Bertrand Russell
    Steven T. Hatton, Jun 24, 2005
    #1
    1. Advertising

  2. Steven T. Hatton wrote:
    > I haven't looked very closely, but from what I'm seeing, it looks like there
    > are no buffered I/O streams in the Standard Library. There are stream
    > buffers, but not buffered streams. [...]
    >
    > Does the standard library not provide a generic buffered stream?
    >


    Uh... So, let me get this straight.... If a stream has a buffer, you still
    do not consider it a "buffered stream"? Huh... Interesting...
    Victor Bazarov, Jun 24, 2005
    #2
    1. Advertising

  3. > I haven't looked very closely
    > I don't have an excellent definition of what a buffered stream is


    If you haven't looked hard for something you don't understand, how can
    you tell if it's there or not?
    James Daughtry, Jun 24, 2005
    #3
  4. Victor Bazarov wrote:

    > Steven T. Hatton wrote:
    >> I haven't looked very closely, but from what I'm seeing, it looks like
    >> there
    >> are no buffered I/O streams in the Standard Library. There are stream
    >> buffers, but not buffered streams. [...]
    >>
    >> Does the standard library not provide a generic buffered stream?
    >>

    >
    > Uh... So, let me get this straight.... If a stream has a buffer, you still
    > do not consider it a "buffered stream"? Huh... Interesting...


    It just wasn't what I had expected. Fact of the matter is, it was far
    simpler than I thought it would be. What I wanted to do was read a file in
    to an in-memory buffer, and then process it as if it were being read from a
    stream. I'm still not sure how to do that with binary data, but then I
    haven't attempted that either. The trick was `ss << in.rdbuf();'. All the
    examples I was seeing were reading one character at a time from cin, or one
    line at at time from a file. I suspect that is not the best way to read
    from system resources.

    #include <fstream>
    #include <iostream>
    #include <sstream>
    #include <string>
    #include <map>

    using namespace std;

    ostream& printPair(ostream& out, const pair<const string, string>& p) {
    return out << "key=" << p.first << ", value=" << p.second << endl;
    }

    template<typename T>
    istream& setPair(istream& in, map<const T,T>& m) {
    string key;
    string value;
    in >> key >> value;
    if(key=="" || value=="") return in;
    m[key] = value;
    return in;
    }


    int main(int argc, char* argv[]) {
    if(!argc > 1) {
    cerr << "rss filename: " << endl;
    return -1;
    }

    ifstream in(argv[1]);

    if(!in.is_open()) {
    cerr << "Failed to open file: " << argv[1] << endl;
    return -1;
    }

    stringstream ss;
    ss << in.rdbuf();
    typedef map<const string, string> Map_T;
    Map_T smap;
    while(setPair(ss, smap));

    for(Map_T::iterator it = smap.begin();it != smap.end(); it++ )
    printPair(cout, *it);
    }

    --
    If our hypothesis is about anything and not about some one or more
    particular things, then our deductions constitute mathematics. Thus
    mathematics may be defined as the subject in which we never know what we
    are talking about, nor whether what we are saying is true.-Bertrand Russell
    Steven T. Hatton, Jun 24, 2005
    #4
  5. Steven T. Hatton wrote:
    > it looks like there
    > are no buffered I/O streams in the Standard Library.


    No, streams are buffered by default. e.g.

    ifstream f("...");
    for(int i=0; i<100; ++i) {
    char c;
    f >> c;
    }

    does NOT do 100 calls to read(), it probably does 1.


    --Phil.
    Phil Endecott, Jun 25, 2005
    #5
  6. Steven T. Hatton

    Rolf Magnus Guest

    Phil Endecott wrote:

    > Steven T. Hatton wrote:
    >> it looks like there
    >> are no buffered I/O streams in the Standard Library.

    >
    > No, streams are buffered by default. e.g.
    >
    > ifstream f("...");
    > for(int i=0; i<100; ++i) {
    > char c;
    > f >> c;
    > }
    >
    > does NOT do 100 calls to read(), it probably does 1.


    As an example, on my linux system, a trace of the system calls (strace)
    shows for the above code:

    open("...", O_RDONLY) = 3
    fstat64(0x3, 0xbffff62c) = 0
    old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
    read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072) = 614
    read(3, "", 131072) = 0
    close(3) = 0

    That means that the implementation reads and buffers the data from files
    in blocks of 128kB. The old_mmap is the allocation of the buffer.
    Rolf Magnus, Jun 25, 2005
    #6
  7. Rolf Magnus wrote:

    > Phil Endecott wrote:
    >
    >> Steven T. Hatton wrote:
    >>> it looks like there
    >>> are no buffered I/O streams in the Standard Library.

    >>
    >> No, streams are buffered by default. e.g.
    >>
    >> ifstream f("...");
    >> for(int i=0; i<100; ++i) {
    >> char c;
    >> f >> c;
    >> }
    >>
    >> does NOT do 100 calls to read(), it probably does 1.

    >
    > As an example, on my linux system, a trace of the system calls (strace)
    > shows for the above code:
    >
    > open("...", O_RDONLY) = 3
    > fstat64(0x3, 0xbffff62c) = 0
    > old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
    > -1, 0) = 0x40018000 read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072)
    > = 614
    > read(3, "", 131072) = 0
    > close(3) = 0
    >
    > That means that the implementation reads and buffers the data from files
    > in blocks of 128kB. The old_mmap is the allocation of the buffer.


    I'd have to read the manpages on the system to understand the output, but
    I'll take your word for it. This behavior is typically what a person would
    want. I'm fairly confident it is not specified by the Standard(though I
    could well be wrong). There are times when this is not what a programmer
    would want. For example using embedded systems with limited memory, or
    very large AV files. I suspect the same might apply when working with
    databases that hold all of their data in a single file. I am certain that
    there were (significant) performance differences between using input
    streams, and buffered input streams in Java when transferring data over a
    network connection. I'm fairly confident the same applies to file access.

    That seems to suggest that Java offers finer grained control over I/O than
    does C++ at the same level of abstraction. That finer control comes at the
    cost of more complexity in the API. Typically we would expect just the
    opposite when conparing Java and C++. I have less than 24 hours experience
    dealing with this topic in C++ (and many of them were spent sleeping) so I
    have much to learn before I can make a meaningful assessment.

    This, BTW, is where I found out how to push file contents from an ifstream
    to a stringstream with one ">>".

    http://www.mindview.net/Books/TICPP/ThinkingInCPP2e.html

    I believe it was Bob Tisdale who suggested Eckel's books as a good source on
    I/O.
    --
    If our hypothesis is about anything and not about some one or more
    particular things, then our deductions constitute mathematics. Thus
    mathematics may be defined as the subject in which we never know what we
    are talking about, nor whether what we are saying is true.-Bertrand Russell
    Steven T. Hatton, Jun 25, 2005
    #7
  8. Steven T. Hatton

    Rolf Magnus Guest

    Steven T. Hatton wrote:

    > Rolf Magnus wrote:
    >
    >> Phil Endecott wrote:
    >>
    >>> Steven T. Hatton wrote:
    >>>> it looks like there
    >>>> are no buffered I/O streams in the Standard Library.
    >>>
    >>> No, streams are buffered by default. e.g.
    >>>
    >>> ifstream f("...");
    >>> for(int i=0; i<100; ++i) {
    >>> char c;
    >>> f >> c;
    >>> }
    >>>
    >>> does NOT do 100 calls to read(), it probably does 1.

    >>
    >> As an example, on my linux system, a trace of the system calls (strace)
    >> shows for the above code:
    >>
    >> open("...", O_RDONLY) = 3
    >> fstat64(0x3, 0xbffff62c) = 0
    >> old_mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
    >> -1, 0) = 0x40018000
    >> read(3, "lakzsdfjlzadfkjaslzdfkjasdflzkja"..., 131072) = 614
    >> read(3, "", 131072) = 0
    >> close(3) = 0
    >>
    >> That means that the implementation reads and buffers the data from files
    >> in blocks of 128kB. The old_mmap is the allocation of the buffer.

    >
    > I'd have to read the manpages on the system to understand the output, but
    > I'll take your word for it.


    I thought it mostly speaks for itself, but forgot that it might not be so
    obvious if you're not so familiar with this kind of system.
    Some explanation:

    The first line says that the file was opened in read-only mode. The returned
    value (the 3 at the end) is the file id.

    The second line just retrieves some information about the file (first
    parameter is the file id, the second a pointer to the struct that the
    function fills). The return value of 0 means success. I'm not sure what the
    stream does with that information.

    The old_mmap in the 3rd line allocates 131072 bytes of memory. The returned
    value is its address. This is the low-level OS interface that malloc and
    new on my system use.

    The next two lines do the actual reading from the file. They each try to
    read 131072 bytes from the file (first parameter is the id, the second one
    a pointer to the memory that the function reads to - the output shows the
    first few bytes actually read, and parameter 3 is the number of bytes to
    read). The returned value of 614 is the number of bytes actually read. The
    second read() returns 0, which indicates nothing more to read, i.e. end of
    file.

    Well, and the last line finally closes the file.

    > This behavior is typically what a person would want. I'm fairly confident
    > it is not specified by the Standard(though I could well be wrong). There
    > are times when this is not what a programmer would want. For example
    > using embedded systems with limited memory, or very large AV files.


    Right, but in such a situation, people typically resort to system specific
    functions. Especially in the second case, you might not only want to switch
    off the in-program buffering, but also open the file with another OS hard
    disk cache strategy. And you have to read in reasonably sized chunks
    yourself then, and the strategy you have to use for is often system
    specific too. So I don't think that a standard stream would help in such a
    situation.
    OTOH, you can implement your own streambuf and use the standard streams with
    it.

    > I suspect the same might apply when working with databases that hold all
    > of their data in a single file. I am certain that there were
    > (significant) performance differences between using input streams, and
    > buffered input streams in Java when transferring data over a network
    > connection. I'm fairly confident the same applies to file access.
    >
    > That seems to suggest that Java offers finer grained control over I/O than
    > does C++ at the same level of abstraction. That finer control comes at
    > the cost of more complexity in the API. Typically we would expect just
    > the opposite when conparing Java and C++. I have less than 24 hours
    > experience dealing with this topic in C++ (and many of them were spent
    > sleeping) so I have much to learn before I can make a meaningful
    > assessment.


    The C++ standard streams and streambuffers don't offer much flexibility in
    this direction.
    Rolf Magnus, Jun 25, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fredrik Lundh
    Replies:
    0
    Views:
    420
    Fredrik Lundh
    Feb 4, 2005
  2. Daniel Bickett
    Replies:
    0
    Views:
    369
    Daniel Bickett
    Feb 4, 2005
  3. steve.leach

    How standard is the standard library?

    steve.leach, Apr 18, 2005, in forum: Python
    Replies:
    1
    Views:
    364
    Christos TZOTZIOY Georgiou
    Apr 18, 2005
  4. funkyj
    Replies:
    5
    Views:
    1,097
    funkyj
    Jan 20, 2006
  5. Replies:
    9
    Views:
    657
    Michael Wojcik
    Aug 23, 2005
Loading...

Share This Page