Newbie: implementing lowlevel protocol over socket int32 data handling

[Bram]

Hi,
Summ: "What is the best way to handle int8,16 and 32 data in python?"

Im currently working on a class to implement the gui protocol of
mldonkey. (http://mldonkey.lemmster.de/wiki/index.php/GuiProtocol)
However this requires to send int32 and int16 and even int8 integers.

To create data I'm using the work-around of converting everything to hex
values, and then converting the hex values to a data string wich I put
on the socket (e.g. "\x00\xF0").
I could do the same on the recieving end: chop it up in bytes and
convert them to numbers (combining the hi and lo bytes of the int16 with
some calculations)

However, there must be a better way of doing this.
Can anyone help me on this problem?

Bram

 

[John Roth]

Hi,
Summ: "What is the best way to handle int8,16 and 32 data in python?"

Im currently working on a class to implement the gui protocol of
mldonkey. (http://mldonkey.lemmster.de/wiki/index.php/GuiProtocol)
However this requires to send int32 and int16 and even int8 integers.

To create data I'm using the work-around of converting everything to hex
values, and then converting the hex values to a data string which I put
on the socket (e.g. "\x00\xF0").
I could do the same on the recieving end: chop it up in bytes and
convert them to numbers (combining the hi and lo bytes of the int16 with
some calculations)

However, there must be a better way of doing this.
Can anyone help me on this problem?

I took a quick look at the Wiki describing the protocol. For
something like this, I'd expect that interoperability would be
an issue, so the actual data format in the packets would be
part of the protocol specification.

That being the case, look at the struct module. I think it does
exactly what you need.

John Roth

 

[Bram]

Thank you for replying so quickly.
You thought right, this resolves all my problems
Bram

 

[Steve Horsley]

This should make it a lot easier for you.

http://www.python.org/doc/current/lib/module-struct.html

Wow! How incredibly useful!
I just had to say that.

Steve

 

[Tero Saarni]

Bram <[email protected]> wrote in message

This should make it a lot easier for you.

http://www.python.org/doc/current/lib/module-struct.html

But this will probably be the 4th post containing such a reference.
As undoubtedly 3 other people have posted it, but have not yet shown
up on google groups.

Instead of the "raw" Struct I was experimenting couple of days ago
with writing something similar to Java ByteBuffer [1]. I use it like this:

---------------------------------
writebuf = ByteBuffer.ByteBufferWriter(file('test.dat', 'wb'))

writebuf.uint8(0x01)
writebuf.uint16(0x0203)
writebuf.string('abc', 4)
writebuf.uint32(0x04050607)
writebuf.flush()

readbuf = ByteBuffer.ByteBufferReader(file('test.dat', 'rb'))

print "uint8=0x%02x" % readbuf.uint8()
print "uint16=0x%04x" % readbuf.uint16()
print "string=%s" % readbuf.string(4)
print "uint32=0x%08x" % readbuf.uint32()
---------------------------------

This code will output:

uint8=0x01
uint16=0x0203
string=abc
uint32=0x04050607

od -t x1c test.dat

0000000 01 03 02 61 62 63 00 07 06 05 04
001 003 002 a b c \0 \a 006 005 004
0000013

The byte order can be changed:
writebuf.order(ByteBuffer.BIG_ENDIAN)

My intention is that the classes take a file like object to
work with so using socket instead of file should be fine.

See the class ByteBuffer.py in the attachment in this post.

Please note that the only test I've executed is the code above
so there is no warranties that the classes will work at all

BR,
Tero

[1] http://java.sun.com/j2se/1.4.1/docs/api/java/nio/ByteBuffer.html



import struct



class ByteBuffer:

NATIVE = '@'
BIG_ENDIAN = '>'
LITTLE_ENDIAN = '<'
NETWORK = '!'

def __init__(self, stream):
self._stream = stream
self._order = ByteBuffer.NATIVE

def order(self, byteorder):
self._order = byteorder
return self



class ByteBufferReader(ByteBuffer):

DEFAULT_BUF_SIZE = 512

def __init__(self, stream, bufsize=DEFAULT_BUF_SIZE):
self._bufsize = max(4, bufsize)
self._buf = ""
self._bufptr = self._bufsize
ByteBuffer.__init__(self, stream)


def _readData(self, byteswanted):
bytesleft = self._bufsize - self._bufptr
if bytesleft < byteswanted:
data = self._buf
self._buf = self._stream.read(self._bufsize)
self._bufsize = len(self._buf)
data += self._buf[:byteswanted - bytesleft]
self._bufptr = byteswanted - bytesleft
else:
data = self._buf[self._bufptr:self._bufptr + byteswanted]
self._bufptr += byteswanted
return data


def int8(self):
return struct.unpack('b', self._readData(1))

def uint8(self):
return struct.unpack('B', self._readData(1))

def int16(self):
return struct.unpack(self._order+'h', self._readData(2))

def uint16(self):
return struct.unpack(self._order+'H', self._readData(2))

def int32(self):
return struct.unpack(self._order+'i', self._readData(4))

def uint32(self):
return struct.unpack(self._order+'I', self._readData(4))

def string(self, len):
return struct.unpack(str(len)+'s', self._readData(len))




class ByteBufferWriter(ByteBuffer):

def __init__(self, stream):
ByteBuffer.__init__(self, stream)


def int8(self, value):
self._stream.write(struct.pack('b', value))
return self

def uint8(self, value):
self._stream.write(struct.pack('B', value))
return self

def int16(self, value):
self._stream.write(struct.pack(self._order+'h', value))
return self

def uint16(self, value):
self._stream.write(struct.pack(self._order+'H', value))
return self

def int32(self, value):
self._stream.write(struct.pack(self._order+'i', value))
return self

def uint32(self, value):
self._stream.write(struct.pack(self._order+'I', value))
return self

def string(self, value, len):
self._stream.write(struct.pack(str(len)+'s', value))
return self


def flush(self):
self._stream.flush()
return self