structs for data transfer?

[Oliver Gerlich]

Hello,
I want to transfer messages between a client and a server (over TCP
sockets). A message consists of a message type (like a message "subject"
, the size of the attached data, and the data itself. The data part
should then be able to contain some information whose layout depends on
the message type...
So now I thought I could define some structs which represent the layout
of the additional information, like this:

typedef struct
{
char versionString[30]
bool paused;
int uptime;
} MsgCoreInfoStruct;

And then I use something like this:

MsgCoreInfoStruct myData;
strcpy(myData.versionString, "Server V1.0");
myData.uptime = getUptime();
myData.paused = false;

Message m;
m.setData( (char*)(&myData) , sizeof(myData) );

The Message object then sends the data to the other side.

Question: When I receive such a message, can I just cast the char*
(which points to the data) into MsgCoreInfoStruct* myRecvData?
And can I then use myRecvData->uptime to get the value I sent out?
And, last but not least , can I use this concept if client and server
are running on different platforms (in this case, Linux and Win)?

I have doubts about this (because of data packing, and because
sizeof(bool) might be inconsistent between compilers...), but I'm not
sure...

Can someone tell me if this concept is right or wrong? Or has someone a
better solution for this problem (maybe something else than structs)?

Thanks in advance,
Oliver Gerlich

 

[Phlip]

I want to transfer messages between a client and a server (over TCP
sockets).

I did that, many many winters ago, and wish I didn't. When you communicate,
TCP is too low-level for data. It can handle the low-level handshaking and
message metadata required, but not all the high-level details. When you need
them, you will find yourself adding packets to your packets.

You are asking "how can I make my program very inflexible for no reason?"

You need to serialize your data into XML, and then transfer that in a higher
level protocol, such as HTTP. After you go with a pre-existing system like
those, they will answer all of your low-level questions, such as how do I
make this string arbitary length, or how do I add a new field, or how do I
change a field's meaning, or how to I interpret a transmission error, etc.

The C languages specify the memory layout of raw structures. Only use them
when you have a real reason not to use a higher level protocol. Premature
optimization is the root of all evil.

 

[John Harrison]

Hello,
I want to transfer messages between a client and a server (over TCP
sockets). A message consists of a message type (like a message "subject"
, the size of the attached data, and the data itself. The data part
should then be able to contain some information whose layout depends on
the message type...
So now I thought I could define some structs which represent the layout of
the additional information, like this:

typedef struct
{
char versionString[30]
bool paused;
int uptime;
} MsgCoreInfoStruct;

And then I use something like this:

MsgCoreInfoStruct myData;
strcpy(myData.versionString, "Server V1.0");
myData.uptime = getUptime();
myData.paused = false;

Message m;
m.setData( (char*)(&myData) , sizeof(myData) );

The Message object then sends the data to the other side.

Question: When I receive such a message, can I just cast the char* (which
points to the data) into MsgCoreInfoStruct* myRecvData?
And can I then use myRecvData->uptime to get the value I sent out?
And, last but not least , can I use this concept if client and server
are running on different platforms (in this case, Linux and Win)?

I have doubts about this (because of data packing, and because
sizeof(bool) might be inconsistent between compilers...), but I'm not
sure...

You are right to doubt.

Can someone tell me if this concept is right or wrong? Or has someone a
better solution for this problem (maybe something else than structs)?

Write some code to convert the struct you want to send into a char array.
Write some code to turn that char array back into a struct. Use the first
piece of code when you send, the second when you recieve. Trying to send
anything more complicated than char arrays between different types of
computers is asking for trouble.

Thanks in advance,
Oliver Gerlich

john

 

[Guest]

I have doubts about this (because of data packing, and because
sizeof(bool) might be inconsistent between compilers...), but I'm not
sure...

Yes you are right
1. You can not be sure not only about bool, but also about byte order(endian
type) and int size.
2. Worst than that, there may be alignament difference into the elements of
the stuct.

What you have to do is to "serialize" data for sending and "deserialize" for
receiving. Doing that means to define some rules about how each data type
is converted into a byte array and back.

Can someone tell me if this concept is right or wrong? Or has someone a
better solution for this problem (maybe something else than structs)?

You can always send a structure safelly, after you have defined a method of
serialization of that struct

If you have time, a good book to read it will be Stevens "Unix Network
Programming" for a clear ideea about sockets.

 

[Guest]

You need to serialize your data into XML, and then transfer that in a
higher level protocol, such as HTTP. After you go with a pre-existing
system like those, they will answer all of your low-level questions, such
as how do I make this string arbitary length, or how do I add a new field,
or how do I change a field's meaning, or how to I interpret a transmission
error, etc.

Today, many people use XML over HTTP or even SOAP for the job
regardles if that is good or bad for the application.
The core ideea to keep in mind is the ammount of comunication and answer
time you want.
If these are not a issue (i.e. you send 2 msg/seccond and a 1/2...1 second
is ok) you can use XML over HTTP. If your client is going to exchange
hundreds or more msg/seccond with the server, and you want a very fast
response, then stay away of XML/HTTP.

About SOAP, my advice is use it only when is mandatory: i.e. when your
software have to comunicate with a given product who speak only SOAP.
Soap have a poor design (just look at Corba for comparation) and the
overhead is unbelivable: A Soap message with XML serialised data for the
struct of OP will waste about 20..50 times (yes, TIMES) as much bandwidth
and will waste about 10..100 times as much processing power on the server
side compared with when you serialize data by yourself in binary endian
independent format.

 

[Phlip]

Write some code to convert the struct you want to send into a char array.
Write some code to turn that char array back into a struct. Use the first
piece of code when you send, the second when you recieve. Trying to send
anything more complicated than char arrays between different types of
computers is asking for trouble.

Oh, and then you must figure out either a length system for each part of the
string, or sentinels and delimeters for the ends of strings. Then you must
figure out how to escape the delimiters if your users type them inside
strings. Then you might need a system to name each data element.

Oh, and then you might localize, and need to pack UTF-8 into your strings.

Use XML. If HTTP is slow, even to a server you programmed, then rewrite a
simpler version of it.

 

[away]

Today, many people use XML over HTTP or even SOAP for the job
regardles if that is good or bad for the application.
The core ideea to keep in mind is the ammount of comunication and answer
time you want.
If these are not a issue (i.e. you send 2 msg/seccond and a 1/2...1 second
is ok) you can use XML over HTTP. If your client is going to exchange
hundreds or more msg/seccond with the server, and you want a very fast
response, then stay away of XML/HTTP.

Why XML/HTTP is slow?

About SOAP, my advice is use it only when is mandatory: i.e. when your
software have to comunicate with a given product who speak only SOAP.
Soap have a poor design (just look at Corba for comparation) and the
overhead is unbelivable: A Soap message with XML serialised data for the
struct of OP will waste about 20..50 times (yes, TIMES) as much bandwidth
and will waste about 10..100 times as much processing power on the server
side compared with when you serialize data by yourself in binary endian
independent format.

What consists overheads of SOAP?

Thanks for your insights in advance!

 

[John Harrison]

Oh, and then you must figure out either a length system for each part of
the
string, or sentinels and delimeters for the ends of strings. Then you must
figure out how to escape the delimiters if your users type them inside
strings. Then you might need a system to name each data element.

Oh, and then you might localize, and need to pack UTF-8 into your strings.

Use XML. If HTTP is slow, even to a server you programmed, then rewrite a
simpler version of it.

Good advice but I got the impression that XML might be a bit beyond the OP.

john

 

[Guest]

Why XML/HTTP is slow?

on a standard x86 compiler
short n=htons(h);
will translate into 2 CPU instruction if optimisation is on

ostringstream str;
str<<"<val>"<<h<<"</val>";
string n=str.str();

will be at least hundreds if no more. It will require heap alocation etc...

in first example sizeof(s) will be 2
into the second s.size() will be at least 12 and at most 16.
If you add the <?xml version="1.0" ..... or namespaces
you got the point.

A dedicated TCP protocol implemented can send messages using only
a minimum overhead (like you can define your own message header with
one byte message type and a long the message size). Just compare the size
of a minimal http header with that. The http being stateless, you may need
to implement your own state keeping procedure if you need that. The
amount of the code executed inside of a application server prior to the
message reaching your handlers can be estimated very conservative to at
least thousands CPU instructions.


For most applications this is OK. If your request is going to make a query
into a database with milions of records, the overhead not significant
compared with the time spent in query. However, for some aplications
it may be unacceptable.

The point I made was NOT that you do not have to use XML. XML is very good
for most of the applications. But you can not state that this is the only
solution, as your answer:
"""
You need to serialize your data into XML, and then transfer that in a higher
level protocol, such as HTTP.
"""

without having a good insight of OP problem.

And by the way, there are over there some modern C++ socket libraries
and with a good implementation of message serialization, a custom socket
protocol may be way easier to implement that J2EE server side solution

What consists overheads of SOAP?

Usually, SOAP just double the amount of data to be send
(serialized/send/deserialized) over a simple POST over HTTP.
Just look at the SOAP envelope required to send a single integer
as specified above

Again, if your application do a query at every 2 seconds over a SQL
database, it do not mater. But at 100+ msg/s a SOAP implementation
will just collapse.

Thanks for your insights in advance!

You are welcome.