Working directly on output stream

[mathieu]

Hi there,

I am trying to rewrite this very slow function (due to AFAIK the
extra copy):


void DoAction(std::istream &is, std:stream &os)
{
uint16_t c;
while( is.read((char*)&c,2) )
{
c =
(c >> (16 - 12 - 1)) & pmask;
os.write((char*)&c, 2 );
}
}

I was thinking of doing:

os.rdbuf( is.rdbuf() )

and then working directly with a ostreambuf_iterator, but I cannot
read from a ostreambuf_iterator can I ?

Thanks for suggestion,
-Mathieu

 

[red floyd]

Hi there,

I am trying to rewrite this very slow function (due to AFAIK the
extra copy):


void DoAction(std::istream &is, std:stream &os)
{
uint16_t c;
while( is.read((char*)&c,2) )
{
c =
(c >> (16 - 12 - 1)) & pmask;
os.write((char*)&c, 2 );
}
}

I was thinking of doing:

os.rdbuf( is.rdbuf() )

and then working directly with a ostreambuf_iterator, but I cannot
read from a ostreambuf_iterator can I ?

Don't reassign the output streambuf. Use transform, or a loop on the
streambuf iterators.

i.e.:

struct convert
{
uint16_t operator()(const uint16 c) const
{
return (c >> shift) & pmask);
}
convert(uint16_t& pmask_) : pmask(pmask_) { }
private:
static const shift = (16 - 12 - 1);
uint16_t pmask;

};

void DoAction(std::istream& is, std:stream& os)
{
std::transform(std::istreambuf_iterator<uint16_t>(is),
std::istreambuf_iterator<uint16_t>(),
std:streambuf_iterator<uint16_t>(os));
}


or

void DoAction(std::istream& is, std:stream& os)
{
std::istreambuf_iterator<uint16_t> in(is);
const istream_iterator<uint16_t> end;
std:streambuf_iterator<uint16_t> out(os);

while (in != end)
*out++ = ((*in++) >> (16 - 12 - 1)) & pmask;
}

 

[red floyd]

Don't reassign the output streambuf. Use transform, or a loop on the
streambuf iterators.

i.e.:

struct convert
{
uint16_t operator()(const uint16 c) const
{
return (c >> shift) & pmask);
}
convert(uint16_t& pmask_) : pmask(pmask_) { }
private:
static const shift = (16 - 12 - 1);

Damn. That should be:
static const unsigned shift = (16 - 12 - 21);

 

[James Kanze]

I am trying to rewrite this very slow function (due to AFAIK the
extra copy):

What makes you think that? I doubt that the extra copy is
measurable compared to the time needed for the I/O itself.

void DoAction(std::istream &is, std:stream &os)
{
uint16_t c;
while( is.read((char*)&c,2) )

You do realize that this is very, very implementation dependent,
and likely not to do what you want with a different compiler.

{
c =
(c >> (16 - 12 - 1)) & pmask;
os.write((char*)&c, 2 );
}
}

I was thinking of doing:

os.rdbuf( is.rdbuf() )

And what on earth is that supposed to do? You're telling the
output stream to use the streambuf from the input stream---in
the most frequent case I can imagine, the streambuf of the input
stream will be a filebuf opened for input, which means all
writes to it will fail.

and then working directly with a ostreambuf_iterator, but I cannot
read from a ostreambuf_iterator can I ?

Of course not.

Not knowing what you're trying to do, it's hard to suggest a
solution.

 

[James Kanze]

Don't reassign the output streambuf. Use transform, or a loop on the
streambuf iterators.

struct convert
{
uint16_t operator()(const uint16 c) const
{
return (c >> shift) & pmask);
}
convert(uint16_t& pmask_) : pmask(pmask_) { }
private:
static const shift = (16 - 12 - 1);
uint16_t pmask;
};

void DoAction(std::istream& is, std:stream& os)
{
std::transform(std::istreambuf_iterator<uint16_t>(is),
std::istreambuf_iterator<uint16_t>(),
std:streambuf_iterator<uint16_t>(os));

And since when can you initialize an
[io]streambuf_iterator<uint16_t> with an [io]stream? [io]stream

}

void DoAction(std::istream& is, std:stream& os)
{
std::istreambuf_iterator<uint16_t> in(is);
const istream_iterator<uint16_t> end;
std:streambuf_iterator<uint16_t> out(os);
while (in != end)
*out++ = ((*in++) >> (16 - 12 - 1)) & pmask;
}

Same problem.

From what little he's posted, he's copying data from an input
stream to an output stream. Both of which are presumably
different (e.g. different files). Which means that a copy will
be necessary. Until he explains what he's really trying to do
(what is the input data, what does he want in output, etc.),
it's hard to say more. He said something about an "extra copy"
in his original posting, but frankly, I don't see one in the
code he posted.

(If he's reading binary data, something like:

uint16_t c = is.get() << 8 ;
c |= is.get() ;
c = (c >> 3) & pmask ;
os.put( c >> 8 ) ;
os.put( c ) ;

in the loop might be faster; a good compiler should be able to
optimize this so that c is always in a register. In practice, I
doubt that the difference will be measurable, however, but this
does have the added advantage of actually having portably
defined semantics, which his original code didn't.)

James Kanze (GABI Software) mailto:[email protected]
Conseils en informatique orient�e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S�mard, 78210 St.-Cyr-l'�cole, France, +33 (0)1 30 23 00 34

 

[mathieu]

Because my first post was not clear, I'll try again to explain what I
am looking for. Let's assume I had access directly to the memory, I
would do:

void DoAction(const char *rawinput, unsigned long l)
{
uint16_t *deb = (uint16_t*)rawinput;
for(unsigned long i = 0; i<l/2; i++)
{
*deb = (*deb >> (16 - 12 - 1)) & pmask;
deb++;
}
}

Well for the design of my IO library I decided that it would be more
convenient (more flexible) to design it with stream, since the stream
could hide a subclass and could hide a compressed stream. So I am
stuck with an API which is:

void DoAction(std::istream& is, std:stream& os);

My previous design (with raw pointer) is a factor two faster than the
current one (with stream interface), and that's a real issue.

In this *very* particular case I know I will not be reusing the input
buffer (input stream) and it has all the good properties so that I can
directly work on the input buffer.

Is there a way to run a filter in-place on the is.rdbuf, and then
simply pass is.rdbuf() to os ?

Hopefully this time it does make sense,

Thanks again,
-Mathieu

 

[red floyd]

Don't reassign the output streambuf. Use transform, or a loop on the
streambuf iterators.

struct convert
{
uint16_t operator()(const uint16 c) const
{
return (c >> shift) & pmask);
}
convert(uint16_t& pmask_) : pmask(pmask_) { }
private:
static const shift = (16 - 12 - 1);
uint16_t pmask;
};

void DoAction(std::istream& is, std:stream& os)
{
std::transform(std::istreambuf_iterator<uint16_t>(is),
std::istreambuf_iterator<uint16_t>(),
std:streambuf_iterator<uint16_t>(os));

And since when can you initialize an
[io]streambuf_iterator<uint16_t> with an [io]stream? [io]stream
is a typedef for basic_[io]stream<char>, which means that it can
only be used with an [io]streambuf_char.

You're right. I was confused. My mistake.

 

[James Kanze]

Because my first post was not clear, I'll try again to explain what I
am looking for. Let's assume I had access directly to the memory, I
would do:

void DoAction(const char *rawinput, unsigned long l)
{
uint16_t *deb = (uint16_t*)rawinput;
for(unsigned long i = 0; i<l/2; i++)
{
*deb = (*deb >> (16 - 12 - 1)) & pmask;
deb++;
}
}

Well for the design of my IO library I decided that it would be more
convenient (more flexible) to design it with stream, since the stream
could hide a subclass and could hide a compressed stream.

Streams aren't particularly designed for modifying memory in
place. As others have mentionned, what you're looking for is
probably std::transform.

You can use stream_iterators with std::transform, but if the
file is binary, you'll have to design some sort of binary stream
for them, and define stream_iterators for it.

So I am stuck with an API which is:

void DoAction(std::istream& is, std:stream& os);

My previous design (with raw pointer) is a factor two faster than the
current one (with stream interface), and that's a real issue.

Certainly: direct in place or memory to memory will almost
always be faster than stream to stream.

In this *very* particular case I know I will not be reusing
the input buffer (input stream) and it has all the good
properties so that I can directly work on the input buffer.

I'm not sure I understand. Do you mean that you know that the
transformation will be in place and in memory?

Is there a way to run a filter in-place on the is.rdbuf, and then
simply pass is.rdbuf() to os ?

Well, you can always use a filtering streambuf. But if your
goal is an in place transformation in memory, any introduction
of streams can only be considered obfuscation.

Hopefully this time it does make sense,

Not really. I doesn't make sense to use streams in the
interface if all you want is an in place transformation.