sending binary files to a 16 micro controller.

[johnny.karlsson]

Hi, I'm working on a project were a need to be able to upload firmware
to a microcontroller based Ethernet device. But because of the memory
constraints the controller can only handle packages of 300 bytes each
time. So therefore the firmware file must be sent in chunks and i need
a header in each file describing which part of the file it is I'm
sending. Could anyone give me some pointer on how a could accomplish
that in python? I'm talking about the client that uploads the software
to the device via TCP.

Best regards

Johnny Karlsson

 

[Magnus Lycka]

Hi, I'm working on a project were a need to be able to upload firmware
to a microcontroller based Ethernet device. But because of the memory
constraints the controller can only handle packages of 300 bytes each
time. So therefore the firmware file must be sent in chunks and i need
a header in each file describing which part of the file it is I'm
sending. Could anyone give me some pointer on how a could accomplish
that in python? I'm talking about the client that uploads the software
to the device via TCP.

You can probably start here:
http://docs.python.org/lib/socket-example.html

If you're uncertain about the exact content of the transmitted
messages, and has access to some other client program, you could
write a Python server and see exactly what bit pattern the other
client sends. Then you can write a Python client that behaves the
same way.

The basic pattern to split and transmit your file would probably
be something like:

f = open('filename')
header_template = 'Chunk %05i, %03i bytes'
for i, bytes in enumerate(f.read(max_number_of_bytes_per_chunk)):
msg = header_template % (i, len(bytes))
msg += bytes
sock.send(msg)
sock.send('The end!')

 

[Magnus Lycka]

header_template = 'Chunk %05i, %03i bytes'

BTW, if the header is binary, you probably want to have a look at
the struct module. http://docs.python.org/lib/module-struct.html

 

[Grant Edwards]

I'm working on a project were a need to be able to upload firmware
to a microcontroller based Ethernet device. But because of the memory
constraints the controller can only handle packages of 300 bytes each
time. So therefore the firmware file must be sent in chunks and i need
a header in each file describing which part of the file it is I'm
sending. Could anyone give me some pointer on how a could accomplish
that in python? I'm talking about the client that uploads the software
to the device via TCP.

You have no control over packet size in TCP if you use the
normal socket interface. About the only thing you can to is
put delays between calls to send() in hope that the TCP stack
will send a packet.

If you really do want control over packet size, you'll have to
use a raw socket and impliment TCP yourself.

 

[Grant Edwards]

Hi, I'm working on a project were a need to be able to upload firmware
to a microcontroller based Ethernet device. But because of the memory
constraints the controller can only handle packages of 300 bytes each
time. So therefore the firmware file must be sent in chunks

[...]

The basic pattern to split and transmit your file would probably
be something like:

f = open('filename')
header_template = 'Chunk %05i, %03i bytes'
for i, bytes in enumerate(f.read(max_number_of_bytes_per_chunk)):
msg = header_template % (i, len(bytes))
msg += bytes
sock.send(msg)
sock.send('The end!')

It's very probable that the TCP stack will combine chunks and
send out full Ethernet frames (4K bytes).

You're probably going to have to either put a time delay in the
loop, or wait for each chunk to be acknowledged before sending
the next one.

 

[johnny.karlsson]

aaah, well i believe that in Windows XPSP2 has disabled raw socket
support (yes i sadly have to use windows) so that's no option. But I'll
try to put a time delay and check what happens. Or otherwise perhaps i
could do a socket.close each time, but that's a terrible waste of
packets.

 

[Grant Edwards]

aaah, well i believe that in Windows XPSP2 has disabled raw socket
support (yes i sadly have to use windows) so that's no option. But I'll
try to put a time delay and check what happens. Or otherwise perhaps i
could do a socket.close each time, but that's a terrible waste of
packets.

I doubt the other end is expecting you to close the socket after
each packet. Unless it was designed to tolerate that, it might
not work. A time delay of a few hundred ms will probably work.
I once wrote a Linux driver that could be used to wait until
the TCP out queue was empty, but that won't do you much good on
Windows.

 

[Richard Brodie]

It's very probable that the TCP stack will combine chunks and
send out full Ethernet frames (4K bytes).

You're probably going to have to either put a time delay in the
loop, or wait for each chunk to be acknowledged before sending
the next one.

Yes, although it would be a rather perverse design from the
controller point of view (it's not clear to me whether that is
a given).

 

[johnny.karlsson]

I'm going to try the timed wait alternative, if i get it the
application to work properly i'll post the results in this group and
the code if anyone want's it. It's such a contrast to code a tcp-server
for the microcontroller (MC9S12NE64) in C and coding in python I
really hope embedded python takes of again!

 

[Peter Hansen]

I'm going to try the timed wait alternative, if i get it the
application to work properly i'll post the results in this group and
the code if anyone want's it. It's such a contrast to code a tcp-server
for the microcontroller (MC9S12NE64) in C and coding in python I
really hope embedded python takes of again!

You mean you wrote the TCP server in the micro in the first place? You
must already know about its restrictions then and, more importantly,
what it expects in the way of time delays and such. Also, couldn't you
have supported UDP as well? It's simpler and on the PC side you can
ensure that small packets are sent without being combined or any such
monkey business.

As for embedded Python, it's very unlikely any Python that might ever be
shoe-horned into an HC12 is going to support enough of Python to make
this a no-brainer. And Python doesn't itself provide the TCP stack, so
even this wouldn't have helped.

-Peter

 

[Nick Craig-Wood]

You have no control over packet size in TCP if you use the
normal socket interface. About the only thing you can to is
put delays between calls to send() in hope that the TCP stack
will send a packet.

You can set the MTU (maximum transfer unit) for that interface. You
do with with ifconfig under un*x - I expect windows has an interface
to do it too (perhaps ipconfig?)

For ethernet the MTU is 1500 bytes normally.

If you really do want control over packet size, you'll have to
use a raw socket and impliment TCP yourself.

Actually I'd recommend the OP uses UDP, not TCP. I've implemented a
few systems which speak UDP directly and its very easy. I wouldn't
like to implement TCP though!

 

[johnny.karlsson]

yeah, i didn't mean running python on the HC12 but using compiled
python on a ARM7 or simular would be nice. There was a project for
embedded python but it seems to have died. Yeah i've been thinking
about using UDP but then i would have to impliment checksums and having
a reliable way to send and acknowlege the chunks sent. But i'll start
tinkering with it tomorow and se what i'll end up with. I allready have
a tcp-server for the device so it would'nt be to much hassle to modify
it. i'm using OpenTCP by the way.

 

[Tom Anderson]

It's very probable that the TCP stack will combine chunks and
send out full Ethernet frames (4K bytes).

Those are *very* full ethernet frames!

It seems to me that if the receiver can only handle a certain amount of
data at a time, this should be reflected in its receive window; it should
never advertise a bigger window than it can handle. if that's the case,
the program should be able to write data as fast as it likes, as the
sending TCP module will throttle it down so it goes out in right-sized
drips. I'm not a TCP expert, though.

Anyway, if you are going the small-writes route, make sure you set the
TCP_NODELAY flag, to turn of Nagleing on the connection. I'm not sure if
anyone mentioned that already.

tom