Byte-oriented streaming

C

Chris Swiedler

Is there a C++ STL or boost class hierarchy for byte streaming similar
to ostream and friends for character streaming? Those classes do not
seem designed to support byte streaming (e.g. they interpret
whitespace and null bytes specially).

thanks,
chris
 
K

kasthurirangan.balaji

Is there a C++ STL or boost class hierarchy for byte streaming similar
to ostream and friends for character streaming? Those classes do not
seem designed to support byte streaming (e.g. they interpret
whitespace and null bytes specially).

thanks,
chris

I may be wrong, i guess "binary" mode of streams should fit you.

Thanks,
Balaji.
 
C

Chris Swiedler

I may be wrong, i guess "binary" mode of streams should fit you.

From what I can tell, it's possible to do binary IO with the standard
stream classes, but (practically speaking) only if you use the read()
and write() methods directly. I'm looking to write a binary streaming
interface which we can use to stream binary data with operator<< and
operator>>, e.g. for writing to sockets, files, in-memory buffers,
etc.

Part of the goal is to allow for user-defined streaming functions so
that someone who writes class Foo can write an operator<<(obytestream
&lhs, Foo& rhs) free function which defines how to stream that class
in binary format. So if for no other reason, it would need to be
different from the std stream classes, so that someone could define an
overload for streaming a class in character format, and another for
streaming the same class in binary format.

It seems to me that C++ needs a binary streaming interface, similar to
but separate from istream / ostream etc. If one doesn't exist I can
create my own, but I want to make sure there's not a standard way to
do it.

thanks,
chris
 
K

kasthurirangan.balaji

From what I can tell, it's possible to do binary IO with the standard
stream classes, but (practically speaking) only if you use the read()
and write() methods directly. I'm looking to write a binary streaming
interface which we can use to stream binary data with operator<< and
operator>>, e.g. for writing to sockets, files, in-memory buffers,
etc.
you may want to look at boost::iostreams or even boost::asio. you
could even consider ace library.
Part of the goal is to allow for user-defined streaming functions so
that someone who writes class Foo can write an operator<<(obytestream
&lhs, Foo& rhs) free function which defines how to stream that class
in binary format. So if for no other reason, it would need to be
different from the std stream classes, so that someone could define an
overload for streaming a class in character format, and another for
streaming the same class in binary format.
Looks like you are looking at persistance. you may want to refer
boost::serialization.
It seems to me that C++ needs a binary streaming interface, similar to
but separate from istream / ostream etc. If one doesn't exist I can
create my own, but I want to make sure there's not a standard way to
do it.

thanks,
chris

AFAIK, operator << should be fine if its in binary mode. If its going
to be a class, then that class needs to define operator <<(overload),
internal implementation using read/write.Ofcourse, there are lots of
design considerations atleast for your requirements.

Thanks,
Balaji.
 
P

Pete Becker

From what I can tell, it's possible to do binary IO with the standard
stream classes, but (practically speaking) only if you use the read()
and write() methods directly. I'm looking to write a binary streaming
interface which we can use to stream binary data with operator<< and
operator>>, e.g. for writing to sockets, files, in-memory buffers,
etc.

operator<< and operator>> do formatted I/O. If you're dealing in raw
bytes, you're doing unformatted I/O. That's what read() and write() are
for. Don't mix idioms: you'll confuse everyone. Use a named function
for your interface.
 
P

Pete Becker

What leads you to believe that?

Well, for one thing, the definition of extractors for builtin types.
They treat whitespace as a delimiter.
Can post a short example of what's not working the way you want it to?

The ostream and istream interfaces are meant to allow whatever kind of
I/O you need, via differing implementations (rdbufs). strstream gives
null bytes special meanings, but stringstream (for example) does not.
Depending on your OS (I'm looking at you, MS), file I/O may require a
binary flag; I've never had to specify it on any flavor of Unix or
Linux, though.

Stream inserters and extractors (operator<< and operator>>) are
designed for formatted I/O, not for unformatted I/O.
 
A

AnonMail2005

Is there a C++ STL or boost class hierarchy for byte streaming similar
to ostream and friends for character streaming? Those classes do not
seem designed to support byte streaming (e.g. they interpret
whitespace and null bytes specially).

thanks,
chris

Take a look at boost iostreams. I think they provide classes as a
starting point to do what you want. At the very least, you can get
ideas
from their implementation.

HTH
 
J

Jeff Schwab

Pete said:
Well, for one thing, the definition of extractors for builtin types.
They treat whitespace as a delimiter.

What's the relevance? Support for formatted I/O does not imply lack of
support for unformatted I/O.
Stream inserters and extractors (operator<< and operator>>) are designed
for formatted I/O, not for unformatted I/O.

Right, so why are you bringing them up? The correct methods for
unformatted I/O are read, write, seekg, tellg, etc.
 
J

James Kanze

On Feb 3, 8:52 pm, (e-mail address removed) wrote: [...]
It seems to me that C++ needs a binary streaming interface,
similar to but separate from istream / ostream etc. If one
doesn't exist I can create my own, but I want to make sure
there's not a standard way to do it.

The problem is that there is no universally established binary
format. Should the binary stream be XDR? BER? or something
else?
 
J

James Kanze

operator<< and operator>> do formatted I/O. If you're dealing in raw
bytes, you're doing unformatted I/O. That's what read() and write() are
for. Don't mix idioms: you'll confuse everyone. Use a named function
for your interface.

That's one point. The other is that istream and ostream are
text formatted streams. If your stream has a binary format,
they aren't appropriate. The standard doesn't provide any
support (at present, but I've not seen any proposals to change
it) for binary formatted streams.
 
J

James Kanze

What leads you to believe that? Can post a short example of what's not
working the way you want it to?
The ostream and istream interfaces are meant to allow whatever
kind of I/O you need, via differing implementations (rdbufs).

That's not true. The abstraction behind ostream and istream is
a stream of text. The streambuf's handle the different sinks
and sources, but have no effect whatsoever on the format.

In addition, in istream at least, the convention is that within
the text, white space is the separator, and that multiple
white space is coalesed into a single separator. You can, in
special cases, inhibit this skipping of white space, but it is
the general rule. (On the other hand, "white space" is locale
dependent, so you can play tricks with the locale. I tend to
see this as abuse, in some way, but others don't.)
strstream gives null bytes special meanings, but stringstream
(for example) does not. Depending on your OS (I'm looking at
you, MS), file I/O may require a binary flag; I've never had
to specify it on any flavor of Unix or Linux, though.

The distinction between binary and text modes in filebuf (and
indirectly in ifstream and ofstream) is only concerned with file
types at the OS level, and how the OS interpretation of "text"
is mapped to the C++ interpretation. Unix and its look-alikes
have only one file type, and for historical reasons, C++ has
adopted it's view of what a text file looks like, so no mapping
is necessary. Unix is rather an exception, here, however.

Note that this "binary" is actually a misnomer, since it only
affects the file mode---the abstraction is still a stream of
text.
 
J

James Kanze

What's the relevance? Support for formatted I/O does not
imply lack of support for unformatted I/O.

There is no such thing as unformatted I/O. All data has a
format.

(He should have said text formatted I/O. They don't do BER
formatting either.)
Right, so why are you bringing them up? The correct methods for
unformatted I/O are read, write, seekg, tellg, etc.

The functions read and write are for pre-formatted text. They
don't change the underlying abstraction of the streams.

I have an ixdrstream and an oxdrstream which does binary
formatting, for example, and I'm sure others have similar
classes, for perhaps other formats. The standard doesn't
support binary formats because there are so many of them, none
of the truly standard (in the sense of being used everywhere---a
lot of them are the subject of a standard somewhere).
 
P

Pete Becker

There is no such thing as unformatted I/O. All data has a
format.

But the C++ standard makes exactly that distinction. There are
functions for formatted input and output, and there are functions for
unformatted input and output.
 
J

James Kanze

On 2008-02-05 05:44:30 -0500, James Kanze <[email protected]> said:
But the C++ standard makes exactly that distinction. There are
functions for formatted input and output, and there are
functions for unformatted input and output.

Those are the words in the standard, but IMHO, they are poorly
chosen. There is input and output which is formatted by the
library functions/operators, and there is input and output which
is "formatted" by the user. The fact that the library function
transmits data transparently and don't format it doesn't mean
that that data doesn't have a format.

(Originally, I used the same words as the standard for this
distinction. It does make sense to speak of "unformatted I/O"
in the sense of "unformatted by this function". But so many
people seem to misunderstand this to mean that the data has no
format that I've decided to be more precise.)

And of course, I've seen code which uses the unformatted input
and output functions in such a way that even the programmer
doesn't know the format, hasn't documented it, and can't
reproduce it on another machine. I guess you could call that
"unformatted" (although I'd use words like incompetent or
unprofessional instead):).
 
C

Chris Swiedler

Yes, what I would like to see is the ability to stream in binary
format. By "stream" I mean overloading operator<< and >> in a way
which will only be used for binary streams. Something like:

class Vector
{
public:
int x,y,z;
};

ostream & operator<<(ostream &lhs, Vector &rhs)
{
lhs << "x: " << rhs.x << " y: " << rhs.y << " z:" << rhs.z;
return lhs;
}

binaryostream & operator<<(binaryostream &lhs, Vector &rhs)
{
lhs << rhs.x << rhs.y << rhs.z;
return lhs;
}

binaryistream & operator>>(binaryistream &lhs, Vector &rhs)
{
lhs >> rhs.x >> rhs.y >> rhs.z;
return lhs;
}

We use something like this currently in our code, but it's all with
custom classes. My question was whether there was a std stream concept
which would be a good base class to use (binaryostream / binaryistream
in the above example), and it seems there isn't.

My feeling is that read() and write() on standard ostreams is too
error prone. In the above example, assuming you're sure that a Vector
is the next piece of data in the stream, it's pretty clear that it
will serialize and unserialize from a stream properly.

The Boost::iostream seems interesting, but it's a pretty large
library, and feels like we'd have to bend our project around it rather
than the other way around.

chris
 
C

coal

Yes, what I would like to see is the ability to stream in binary
format. By "stream" I mean overloading operator<< and >> in a way
which will only be used for binary streams. Something like:

class Vector
{
public:
        int x,y,z;

};

ostream & operator<<(ostream &lhs, Vector &rhs)
{
        lhs << "x: " << rhs.x << " y: " << rhs.y << " z:" << rhs.z;
        return lhs;

}

binaryostream & operator<<(binaryostream &lhs, Vector &rhs)
{
        lhs << rhs.x << rhs.y << rhs.z;
        return lhs;

}

binaryistream & operator>>(binaryistream &lhs, Vector &rhs)
{
        lhs >> rhs.x >> rhs.y >> rhs.z;
        return lhs;

}

We use something like this currently in our code, but it's all with
custom classes. My question was whether there was a std stream concept
which would be a good base class to use (binaryostream / binaryistream
in the above example), and it seems there isn't.

My feeling is that read() and write() on standard ostreams is too
error prone. In the above example, assuming you're sure that a Vector
is the next piece of data in the stream, it's pretty clear that it
will serialize and unserialize from a stream properly.

The Boost::iostream seems interesting, but it's a pretty large
library, and feels like we'd have to bend our project around it rather
than the other way around.

I think it is called Boost::iostreams. It probably should have
been called biostreams or something else as it is confusing.

I think the suggestion earlier to use a named function makes
sense and that's what I've done.

Brian Wood
Ebenezer Enterprises
www.webebenezer.net
 
J

James Kanze

Yes, what I would like to see is the ability to stream in binary
format. By "stream" I mean overloading operator<< and >> in a way
which will only be used for binary streams. Something like:
class Vector
{
public:
int x,y,z;
};
ostream & operator<<(ostream &lhs, Vector &rhs)
{
lhs << "x: " << rhs.x << " y: " << rhs.y << " z:" << rhs.z;
return lhs;
}
binaryostream & operator<<(binaryostream &lhs, Vector &rhs)
{
lhs << rhs.x << rhs.y << rhs.z;
return lhs;
}
binaryistream & operator>>(binaryistream &lhs, Vector &rhs)
{
lhs >> rhs.x >> rhs.y >> rhs.z;
return lhs;
}
We use something like this currently in our code, but it's all
with custom classes. My question was whether there was a std
stream concept which would be a good base class to use
(binaryostream / binaryistream in the above example), and it
seems there isn't.

Sort of. My own xdrstream derive from ios (actually,
basic_ios<char>), to get the error handling and the management
of streambuf from the standard streams. ios also contains a
lot of formatting information which is irrelevant to a binary
stream, but it seems better to use it anyway, and offer exactly
the same conventions as the text streams with regards to error
handling. And of course, streambuf doesn't really care about
the format.

Logically, you'd like a stripped down ios and streambuf, without
the formatting flags and the locale, but it doesn't seem worth
the effort to implement them from scratch. (Do document,
however, that if the user provides his own filebuf, it must be
opened in binary mode, and imbued with the "C" locale. Of
course, my derived fxdrstream ensure this with the filebuf they
create.)
My feeling is that read() and write() on standard ostreams is
too error prone.

And how, since it gives you a more or less random and
undocumented binary format. That's not what they're there for;
otherwise, they'd take void* for argument, rather than charT*.
They're present for the case when you have pre-formatted text.
In the above example, assuming you're sure
that a Vector is the next piece of data in the stream, it's
pretty clear that it will serialize and unserialize from a
stream properly.

What will serialize and unserialize properly? Your code, yes.
read and write, no.
The Boost::iostream seems interesting, but it's a pretty large
library, and feels like we'd have to bend our project around
it rather than the other way around.

I've not looked at all of boost::iostream, but the parts I did
look at didn't seem to be oriented binary instead of text. It
was more a question of making it easier to extend text handling.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,578
Members
45,052
Latest member
LucyCarper

Latest Threads

Top