Converting between vectors

[Andrea Crotti]

I need in short to serialize vector of different types into a stream of
chars, which is then sent over the network.
Boost unfortunately is not the answer since I can't use it in the
project.

Until now I used the function below, but I found out that is terribly
buggy, since it doesn't work in all cases of type conversion and even
size.

But I can't find a way to fix it and to make it general, any idea /hint?

Thanks,
Andrea


#include <assert.h>
#include <string>
#include <cstdlib>
#include <iostream>
#include <vector>

template <typename INP, typename OUT>
std::vector<OUT> vectorConversion(const std::vector<INP>& input)
{
size_t size_inp = sizeof(INP);
size_t size_out = sizeof(OUT);
// make sure the sizeof are multiples between them, otherwise it doesn't work!
assert(((size_inp % size_out) == 0) || ((size_out % size_inp) == 0));
std::vector<OUT> res;
size_t size = (size_inp * input.size()) / size_out;
res.resize(size);
// is it memcpy a good idea in this case?
memcpy(&res[0], &input[0], size_inp * input.size());
return res;
}

int main()
{
std::vector<int> vec(5, 1);
std::vector<long> lvec = vectorConversion<int, long>(vec);
std::vector<int> vec2 = vectorConversion<long, int>(lvec);

assert(vec == vec2);
return 0;
}

 

[Victor Bazarov]

I need in short to serialize vector of different types into a stream of
chars, which is then sent over the network.
Boost unfortunately is not the answer since I can't use it in the
project.

Until now I used the function below, but I found out that is terribly
buggy, since it doesn't work in all cases of type conversion and even
size.

But I can't find a way to fix it and to make it general, any idea /hint?

Thanks,
Andrea


#include<assert.h>
#include<string>
#include<cstdlib>
#include<iostream>
#include<vector>

template<typename INP, typename OUT>
std::vector<OUT> vectorConversion(const std::vector<INP>& input)
{
size_t size_inp = sizeof(INP);
size_t size_out = sizeof(OUT);
// make sure the sizeof are multiples between them, otherwise it doesn't work!
assert(((size_inp % size_out) == 0) || ((size_out % size_inp) == 0));
std::vector<OUT> res;
size_t size = (size_inp * input.size()) / size_out;
res.resize(size);
// is it memcpy a good idea in this case?
memcpy(&res[0],&input[0], size_inp * input.size());
return res;
}

int main()
{
std::vector<int> vec(5, 1);
std::vector<long> lvec = vectorConversion<int, long>(vec);
std::vector<int> vec2 = vectorConversion<long, int>(lvec);

assert(vec == vec2);
return 0;
}

First off, you should allow the compiler to detemine the 'INP' type from
the argument. Redefine your template as

template<typename OUT, typename INP> ...

and use it this way:

std::vector<long> lvec = vectorConversion<long>(vec);

then the argument you didn't explicitly provide will be deduced.

Second, 'memcpy' is not a good idea. A loop with assignment from one to
the other should do just fine.

Third, try plain initialization from a pair of iterators:

std::vector<long> lvec(vec.begin(), vec.end());

I am not sure it should work if the value_type types are different, but
it costs next to nothing to try.

V

 

[Andrea Crotti]

First off, you should allow the compiler to detemine the 'INP' type
from the argument. Redefine your template as

template<typename OUT, typename INP> ...

and use it this way:

std::vector<long> lvec = vectorConversion<long>(vec);

then the argument you didn't explicitly provide will be deduced.

Second, 'memcpy' is not a good idea. A loop with assignment from one
to the other should do just fine.

Third, try plain initialization from a pair of iterators:

std::vector<long> lvec(vec.begin(), vec.end());

I am not sure it should work if the value_type types are different,
but it costs next to nothing to try.

V

Ok thanks, I tried the plain initialization but it doesn't work...

But also I don't see how the loop can work, I mean a long doesn't fit in
a char, and on the other way I don't want to use one long for each char,
I would waste a lot of space.

What could be another way to do it?

 

[Andrea Crotti]

A simpler question while I try to figure out how to do this:
How do I make 2 ints fit in one long and then extract them again?

This very rough code doesn't work as expected:

--8<---------------cut here---------------start------------->8---
long intsToLong(int x, int y)
{
long r = (x << sizeof(int));
r += y;
return r;
}

void longToInts(long l)
{
int second = l >> sizeof(int);
int first = (l & (2 << sizeof(int)));
std::cout << " first " << first << " second " << second << std::endl;
}

int main()
{
long l = intsToLong(10, 20);
std::cout << l << " which gives " << std::endl;
longToInts(l);
return 0;
}
--8<---------------cut here---------------end--------------->8---

 

[LR]

I need in short to serialize vector of different types into a stream of
chars, which is then sent over the network.
Boost unfortunately is not the answer since I can't use it in the
project.

Until now I used the function below, but I found out that is terribly
buggy, since it doesn't work in all cases of type conversion and even
size.

Can you please be more specific?

But I can't find a way to fix it and to make it general, any idea /hint?

Thanks,
Andrea


#include <assert.h>
#include <string>
#include <cstdlib>
#include <iostream>
#include <vector>

template <typename INP, typename OUT>
std::vector<OUT> vectorConversion(const std::vector<INP>& input)
{
size_t size_inp = sizeof(INP);
size_t size_out = sizeof(OUT);
// make sure the sizeof are multiples between them, otherwise it doesn't work!
assert(((size_inp % size_out) == 0) || ((size_out % size_inp) == 0));
std::vector<OUT> res;
size_t size = (size_inp * input.size()) / size_out;

It looks to me like you want to pack or unpack the values in input to
res. If that's the case the above line will not work in all cases.
Consider if size_inp == 2 and input.size() == 3 and size_out == 4.

res.resize(size);
// is it memcpy a good idea in this case?
memcpy(&res[0], &input[0], size_inp * input.size());
return res;
}

LR

 

[LR]

A simpler question while I try to figure out how to do this:
How do I make 2 ints fit in one long and then extract them again?

This very rough code doesn't work as expected:

What happens?

--8<---------------cut here---------------start------------->8---
long intsToLong(int x, int y)
{
long r = (x << sizeof(int));

On the platform I use, sizeof(int) == 4, so you're shifting x by 4.

r += y;

I suspect that line will cause some problems for negative numbers.

return r;
}

void longToInts(long l)
{
int second = l >> sizeof(int);
int first = (l & (2 << sizeof(int)));

I'm not sure what the intent of that line is.

std::cout << " first " << first << " second " << second << std::endl;
}

int main()
{
long l = intsToLong(10, 20);
std::cout << l << " which gives " << std::endl;
longToInts(l);
return 0;
}
--8<---------------cut here---------------end--------------->8---

What are sizeof(int) and sizeof(long) on the platform you're using? Does
this code have to be portable? You may find CHAR_BIT of use.

 

[LR]

I need in short to serialize vector of different types into a stream of
chars, which is then sent over the network.
Boost unfortunately is not the answer since I can't use it in the
project.

Until now I used the function below, but I found out that is terribly
buggy, since it doesn't work in all cases of type conversion and even
size.

But I can't find a way to fix it and to make it general, any idea /hint?

Thanks,
Andrea


#include <assert.h>
#include <string>
#include <cstdlib>
#include <iostream>
#include <vector>

template <typename INP, typename OUT>
std::vector<OUT> vectorConversion(const std::vector<INP>& input)
{
size_t size_inp = sizeof(INP);
size_t size_out = sizeof(OUT);
// make sure the sizeof are multiples between them, otherwise it doesn't work!
assert(((size_inp % size_out) == 0) || ((size_out % size_inp) == 0));

I neglected to mention that you might want to check to see if INP and
OUT are integer types using std::numeric_limits::is_integer.

std::vector<OUT> res;
size_t size = (size_inp * input.size()) / size_out;
res.resize(size);
// is it memcpy a good idea in this case?
memcpy(&res[0], &input[0], size_inp * input.size());
return res;
}

 

[Noah Roberts]

Boost unfortunately is not the answer since I can't use it in the
project.

Still.... Use boost.

 

[Victor Bazarov]

A simpler question while I try to figure out how to do this:
How do I make 2 ints fit in one long and then extract them again?
[..]

There is no guarantee it can be done. You've a better chance of
actually achieving that with two shorts and a long, although it is not
guaranteed either since the ranges of values of types are not limited
from above.

V

 

[Ian Collins]

A simpler question while I try to figure out how to do this:
How do I make 2 ints fit in one long and then extract them again?

You can't portably. You would have to use fixed width types.

I'd start with a compile time "assert" like

const bool checkSizes = 1/(sizeof(long)==2*sizeof(int));

followed by

const size_t intBits = sizeof(int)*CHAR_BIT;

This very rough code doesn't work as expected:

--8<---------------cut here---------------start------------->8---
long intsToLong(int x, int y)
{
long r = (x<< sizeof(int));

Use intBits and check the type you are shifting.

r += y;
return r;
}

void longToInts(long l)
{
int second = l>> sizeof(int);

Use intBits.

int first = (l& (2<< sizeof(int)));

(2<< sizeof(int))? What value do you expect this to have? Use a
temporary variable and check its value.

 

[Andrea Crotti]

Can you please be more specific?

The bugs are exactly what you point out later, I never noticed because
the sizes where always such that it always worked anyway...

It looks to me like you want to pack or unpack the values in input to
res. If that's the case the above line will not work in all cases.
Consider if size_inp == 2 and input.size() == 3 and size_out == 4.

Yes I want to do that, any suggestions how to do it more generally and
in a more correct way?

 

[Andrea Crotti]

I neglected to mention that you might want to check to see if INP and
OUT are integer types using std::numeric_limits::is_integer.

Why should I check? In general I should not care to check...
Actually the whole conversion thing only happens from

type T -> unsigned char
and viceversa, no I can always suppose that the input or the output has
size 1 (not sure it makes a difference).

 

[Andrea Crotti]

Still.... Use boost.

I think I should add a signature "I can't use Boost".
Is not that I don't want to, but the strict requirement is that I can
only use the standard library, and actually considered where it will
deployed it makes perfectly sense.

Boost is great, but suggesting it when someone already said that is not
an option doesn't help much unfortunately...

 

[Andrea Crotti]

Could you at least hint about about the kind of environment where it
will be employed where it makes perfect sense to not use Boost?

Network routers in an experimental testbed, asking new software there
is not easy and surely takes too long for the time I have left.
And it was a decision taken in the beginning, so discuss about it is
just a waste of time...

huh.

Well I think I wrote 200 times "I can't use boost", that's why

 

[Andrea Crotti]

You can't portably. You would have to use fixed width types.

I'd start with a compile time "assert" like

const bool checkSizes = 1/(sizeof(long)==2*sizeof(int));

followed by

const size_t intBits = sizeof(int)*CHAR_BIT;

This is just a toy example but I wanted to understand how to do it
correctly...

Like this it seems to work fine, is that correct in theory?
Are there other ways to achieve the same thing in general?

--8<---------------cut here---------------start------------->8---
long intsToLong(int x, int y)
{
long tmp = x;
tmp <<= int_bits;
tmp += y;
return tmp;
}

void longToInts(long l)
{
int second = l >> int_bits;
long mask = (2 >> (int_bits + 1)) - 1;
int first = l & mask;
std::cout << " first " << first << " second " << second << std::endl;
}
--8<---------------cut here---------------end--------------->8---

 

[Alf P. Steinbach /Usenet]

* Andrea Crotti, on 22.04.2011 11:27:

I think I should add a signature "I can't use Boost".
Is not that I don't want to, but the strict requirement is that I can
only use the standard library, and actually considered where it will
deployed it makes perfectly sense.

Could you at least hint about about the kind of environment where it will be
employed where it makes perfect sense to not use Boost?

Boost is great, but suggesting it when someone already said that is not
an option doesn't help much unfortunately...

huh.


Cheers

- Alf

 

[Ian Collins]

This is just a toy example but I wanted to understand how to do it
correctly...

Like this it seems to work fine, is that correct in theory?
Are there other ways to achieve the same thing in general?

Mask and shift is the idiomatic, portable, solution.

--8<---------------cut here---------------start------------->8---
long intsToLong(int x, int y)
{
long tmp = x;
tmp<<= int_bits;
tmp += y;
return tmp;
}

void longToInts(long l)
{
int second = l>> int_bits;
long mask = (2>> (int_bits + 1)) - 1;

Have you by any chance looked at the value of mask? Hint - what is 2
left shifted 31 places?

 

[Andrea Crotti]

Have you by any chance looked at the value of mask? Hint - what is 2
left shifted 31 places?

Yes sure I was shifting in the wrong direction, thanks

So and what if the size of the types is not multiple of each other?
For example type X is 6 bytes and type Y is 4 bytes.

Then what is the best way to pack data from one type to the other?

 

[LR]

The bugs are exactly what you point out later, I never noticed because
the sizes where always such that it always worked anyway...


Yes I want to do that, any suggestions how to do it more generally and
in a more correct way?

I think you'll have to fix the math problem and also you'll want to
maintain the number of characters you're converting.

Personally, I don't think memcpy is a bad way to do this, but perhaps
there is a better way.

LR

 

[LR]

Why should I check? In general I should not care to check...
Actually the whole conversion thing only happens from

type T -> unsigned char
and viceversa, no I can always suppose that the input or the output has
size 1 (not sure it makes a difference).

This makes me feel a little uncertain as to the problem you actually
want to solve, since what you posted converts packs and unpacks long and
int.

LR