Serialization

[Andrea Crotti]

I'm doing a very complicated structure for serialize/deserialize
objects, in short I have this abstract class

--8<---------------cut here---------------start------------->8---
#ifndef STREAMABLE_H
#define STREAMABLE_H

#include "Stream.hpp"

template <typename T>
class Serializable
{
public:
Serializable() {}
virtual Stream toStream() = 0;
// used to parse the given stream and create the object
virtual T parseStream(const Stream&) = 0;
};

#endif /* STREAMABLE_H */
--8<---------------cut here---------------end--------------->8---

and all the objects that want to bea able to create a stream and parse
it have to overload it.

Now 2 problems:
- the definition of the classes is something like
class PadNodeID : public Serializable<PadNodeID>

so the template parameter is always the class itself, maybe is it
possible to avoid it?

- the "parseStream" function doesn't really make sense to be called from
one object, since it's supposed to create one.
But making it static and virtual doens't work, so how should I declare
it in such a way that I can do.

--8<---------------cut here---------------start------------->8---
Type t = Type:arseStream(st);
--8<---------------cut here---------------end--------------->8---

Thanks again,
Andrea

 

[Andrea Crotti]

Why use templates?

class Serializable
{
public:
Serializable() {}
virtual Stream toStream() = 0;
// used to parse the given stream and create the object
virtual void parseStream(Stream&) = 0;
};

class PadNodeID : public Serializable
{
PadNodeID(Stream& aStream) { parseStream(aStream); }
virtual Stream toStream() { ... }
virtual void parseStream(const Stream&) { ... }
};

int main()
{
...
Stream st;
PadNodeID node(st);
}

/Leigh

well actually only because parseStream should be
--8<---------------cut here---------------start------------->8---
virtual T parseStream(const Stream&)
--8<---------------cut here---------------end--------------->8---
where T is the generic type.
Unless I have a way to write the type of the current object directly I
doin't see how else I can do...

About the other question I'm still stuck, rephrasing:
is there a way to have an abstract function which MUST be implemented as
static in every subclass??

 

[Larry Evans]

I'm doing a very complicated structure for serialize/deserialize
objects,

[snip]
Hi Andrea,

I've never used it, but boost has a serialization library:

http://www.boost.org/doc/libs/1_44_0/libs/serialization/doc/index.html

which you may find useful.

-regards,
Larry

 

[Brian Wood]

Why use templates?

class Serializable
{
public:
Serializable() {}
virtual Stream toStream() = 0;
// used to parse the given stream and create the object
virtual void parseStream(Stream&) = 0;

};

class PadNodeID : public Serializable
{
PadNodeID(Stream& aStream) { parseStream(aStream); }
virtual Stream toStream() { ... }
virtual void parseStream(const Stream&) { ... }

};

int main()
{
...
Stream st;
PadNodeID node(st);

}

What is gained by the Serializable base class?
I'd write it like this:


class PadNodeID
{
protected:
void SendMemberData(SendCompressedBuffer* buf) const;

public:
template <typename R>
explicit PadNodeID(ReceiveCompressedBuffer<R>* buf);

void CalculateMarshallingSize(Counter& cntr) const;

void
Send(SendCompressedBuffer* buf, bool = false) const
{
SendMemberData(buf);
}
};


Brian Wood
Ebenezer Enterprises
http://webEbenezer.net

 

[Andrea Crotti]

What is gained by the Serializable base class?
I'd write it like this:


class PadNodeID
{
protected:
void SendMemberData(SendCompressedBuffer* buf) const;

public:
template <typename R>
explicit PadNodeID(ReceiveCompressedBuffer<R>* buf);

void CalculateMarshallingSize(Counter& cntr) const;

void
Send(SendCompressedBuffer* buf, bool = false) const
{
SendMemberData(buf);
}
};

Nothing in practice, but it's an interface that when implemented would
force the programmer to avoid implementing some needed functions.

Of course I can also just implement those functions and that's it.
But I like this

--8<---------------cut here---------------start------------->8---
template <typename R>
explicit PadNodeID(ReceiveCompressedBuffer<R>* buf);
--8<---------------cut here---------------end--------------->8---

Why do you use a pointer here instead of a reference?
And I finally only read now about the explicit, which if I understand
force you to not write something like

PadNodeID p = buffer;

with automatic conversion, but is that really useful or just something
nice to have in these situations?

 

[Andrea Crotti]

I'm doing a very complicated structure for serialize/deserialize
objects,

[snip]
Hi Andrea,

I've never used it, but boost has a serialization library:

http://www.boost.org/doc/libs/1_44_0/libs/serialization/doc/index.html

which you may find useful.

-regards,
Larry

Yes it does look very nice, but the problem is that I can't really use
it.
Or well I can use it for testing and other things but in "production" I
must be able to disable it, and then it's a bit problematic.

I'm not expert in this field at all, I'm looking around and what I only
need is just for example

- take a std::vector<string> var;
- write it in a portable manner
- send and reconstruct

So now suppose that
coord_t = int
and
stream_t = char

--8<---------------cut here---------------start------------->8---
For one of the types I did this
Stream PadCoordinate::toStream()
{
// the stream should also accept a vector of something else
// or a generic iterator, to make life easier to the other part of the code
vector<stream_t> vec;
for (size_t i=0; i < coord.size(); ++i) {
vec.push_back((stream_t) coord);
}
Stream st(vec);
return st;
}

// maybe this conversion is not really a good idea
PadCoordinate PadCoordinate:arseStream(const Stream& raw)
{
// instead of pushing I have to set the values!!
PadCoordinate pc;
for (int i=0; i < raw.getSize(); ++i) {
pc.coord = (coord_t) raw;
}
return pc;
}
--8<---------------cut here---------------end--------------->8---

where Stream is this thing in the end

--8<---------------cut here---------------start------------->8---
class Stream
{
private:
bool stream_is_const;
std::vector<stream_t> stream;
void setStream(stream_t *buf, const int);
...
--8<---------------cut here---------------end--------------->8---

because in the end the data to the other layers must go in
(char *buffer, int *size) format.

But here it was easy since it's easily convertible (even if probably
wrong already since int is much larger).

But what if I have a bigger type?
Does it make sense to use the vector of chars there at all?

Any example of a simple serialization in c++?

 

[Larry Evans]

Nothing in practice, but it's an interface that when implemented would
force the programmer to avoid implementing some needed functions.

[snip]

*Maybe* Brian was thinking that you'd just duplicate this constructor
pattern for each serializable class. IOW, suppose you have an
enumeration tag used to specialize each serialization class where
each specialization would have the needed functions. The following
pseudo uses variadic templates notation:
http://www.osl.iu.edu/~dgregor/cpp/variadic-templates.html
However, I think it should be obvious how to translate
this pseudo code into non-variadic code):

enum serialization_tag
{ t1
, t2
...
, tn
}

template
< typename... T

>

struct pack
/**@brief
* Only used to gather types together.
* See below for examples.
*/
{};

template
< serialization_tag Stag

>

struct tag_serializable
/**@brief
* Most general template is empty
*/
{
public:
template <typename R>
explicit tag_serializable(R)
{}
};

template
< serialization_tag Stag
, typename... Supers
, typename... Members

>

struct tag_serializable
< t1 //specialize on this tag.

>

: Supers...
{
public:
Members...
members
/**@brief
* Like:
* T... members;
* in Douglas Gregor's post:
* See
http://groups.google.com/group/comp.std.c++/msg/40705c1e2a6f78f8
*/
;

template <typename R>
explicit tag_serializable(R r)
: Supers(r)...
, members(r)...
/**@Requirements:
* All Supers... and Members...
* must have a CTOR taking R r as arg.
*
**This CTOR code would be duplicated for each
* serialization_tag, or, if tags were not
* used, then for each class needed to be
* serialized.
*
*/
{}

/*
* The "needed functions"
* (as mentioned On 11/05/10 04:49 by Andrea Crotti.)
* for the t1 specialization would go here.
*/
...
};

The key requirement is (as mentioned in the pseudo-code comments)
All super classes and member variables must have a CTOR with
signature:

(R r)

Since:

: Supers(r)...
, members(r)...

calls the Super... and members... CTOR's in the proper order, the r
would be passed to each one and updated accordingly. IOW, if
R = std::istream, then r would be advanced properly during each CTOR
so that the next CTOR call would have the r positioned at the correct
location in the R. Unfortunated, as noted by Douglas Gregor's post:

Members... members;

is ambiguous and not implemented in the current g++. However, the
alternative is using boost::fusion::vector from variadic fusion:

http://svn.boost.org/svn/boost/sandbox/SOC/2009/fusion/

or

http://svn.boost.org/svn/boost/sandbox/variadic_templates/boost
/composite_storage/pack/container_all_of_aligned.hpp

WARNING. Not tested with any of the above suggested alternatives.

HTH.

-Larry

 

[Brian Wood]

Nothing in practice, but it's an interface that when implemented would
force the programmer to avoid implementing some needed functions.

Of course I can also just implement those functions and that's it.
But I like this

--8<---------------cut here---------------start------------->8---
template <typename R>
explicit PadNodeID(ReceiveCompressedBuffer<R>* buf);
--8<---------------cut here---------------end--------------->8---

Why do you use a pointer here instead of a reference?

I agree a reference would be better and plan to change it.
Thanks for commenting on it.

 

[Ebenezer]

I agree a reference would be better and plan to change it.
Thanks for commenting on it.- Hide quoted text -

- Show quoted text -

This change is now available on line and most of the
on line examples have been updated as well.

 

[Andrea Crotti]

This change is now available on line and most of the
on line examples have been updated as well.

Very good thanks, but I still have a question.
What's the right way to convert generic types to a
std::vector<unsigned char>?

I mean should I maybe
- compute the size
- allocate the right space on the vector considering that every element
is just one byte
- memcpy (should be contigous the memory)

Or is there another way?
If for example I want to serialize a

vector<long> -> vector<char>

vector<char> convert(vector<long>& l)
{
vector<char> res;
size_t size = sizeof(long) * l.size();
res.resize(size);
memcpy(&res, &l, size);
return res;
}

Or am I getting into troubles with something like this??

 

[Andrea Crotti]

I did the following stupid test
typedef unsigned char stream_t;

// maybe I can even use a pointer instead
template <typename T>
vector<stream_t> vectorToStream(vector<T>& input)
{
vector<stream_t> res;
size_t size = sizeof(T) * input.size();
// I'm sure it's always 1
res.resize(size);
// first convert to network order and then copy brutally
memcpy(&res[0], &input[0], size);
return res;
}

// in theory I don't know what type it is??
vector<int> unStream(vector<stream_t>& input)
{
// size_t size = input.size() / sizeof(int);
vector<int> res(input.size());
memcpy(&res[0], &input[0], input.size());

return res;
}

int main(int argc, char *argv[])
{
vector<int> var(10, 1);
vector<stream_t> tmp = vectorToStream<int>(var);
vector<int> res = unStream(tmp);

for (int i = 0; i < 10; ++i) {
assert(var == res);
}

return 0;
}

And it seems to work, in theory I have a generic "Streamer" for vectors
and one inverse operation for vector<int>.

Still not quite sure is the best (or the only) way to go though,
suggestions are welcome