In a bit of a pickle here :)

[Larry goodman]

Hi, Im an old C++ salt who is doing his first large project using
python/wxPython. Fantastically productive are python and wxWindows.
If you could bear with me, I have a couple of questions for people who
have implemented python projects with similar characterstics. I'm
building a client/server app where the client is a windows or linux
desktop and the back end is pyhton middleware (im going to write)
running on linux with a postgres SQL back end.

I started out using the typical client/server approach starting with
my data model. The more i've used python, the more i've gotten to
think I really dont need a relational data model at all. All the
middlware does is serve up pickled python objects to my desktop via
sockets. The desktop model contains all of the logic as my app runs
in an offline briefcase model. So I came up with the idea of just
storing all of the python objects in the postgres database pickled in
a blob field. Im sure about ten thousand other people came up with
this idea before me. The only other data stored in the table will be
the id of the object and the last time the object was modified. I may
also store a CRC for resolution conflict. For instance, if a client
tries to change an object that was also changed by someone else since
they last received it, the server would throw an exception. Are there
any holes with this approach? Is there a better approach? I have to
stress that I cannot use remoting like pyro because the laptop will be
offline most of the day. They will resync with the server at most a
few times a day.

I have one other question about the pickle system. If I add new
attributes to an object and try to unpickle an old version will it
work? How do you handle versioning of objects with pickle?

My last questions involve using sockets as a transport. If I use
python to exchange data via sockets on the server, will my server be
susceptible to buffer overflow attacks? Because I may need to support
handhelds with no SSL capability, I may need to expose a socket to the
internet unsecured. Any idea what the best approach would be to
keeping the bad people out in this instance? How should I secure my
middleware if I cannot support SSL?

If I have to I will only support SSL or running sockets over SSH.

Thanks so much for helping out.

 

[anton muhin]

I started out using the typical client/server approach starting with
my data model. The more i've used python, the more i've gotten to
think I really dont need a relational data model at all. All the
middlware does is serve up pickled python objects to my desktop via
sockets. The desktop model contains all of the logic as my app runs
in an offline briefcase model. So I came up with the idea of just
storing all of the python objects in the postgres database pickled in
a blob field. Im sure about ten thousand other people came up with
this idea before me. The only other data stored in the table will be
the id of the object and the last time the object was modified. I may
also store a CRC for resolution conflict. For instance, if a client
tries to change an object that was also changed by someone else since
they last received it, the server would throw an exception. Are there
any holes with this approach? Is there a better approach? I have to
stress that I cannot use remoting like pyro because the laptop will be
offline most of the day. They will resync with the server at most a
few times a day.

I have one other question about the pickle system. If I add new
attributes to an object and try to unpickle an old version will it
work? How do you handle versioning of objects with pickle?

Warning: I have no experience with such a kind of problems.

If I need to synchronize some data, I'd rather look at version control
systems. IMHO, subversion or cvs seems good enough for this kind of
work. Subversion even have Python bindings (but I didn't play with them).

hope this helps,
anton.

 

[John E. Barham]

Hi, Im an old C++ salt who is doing his first large project using
python/wxPython. Fantastically productive are python and wxWindows.
If you could bear with me, I have a couple of questions for people who
have implemented python projects with similar characterstics. I'm
building a client/server app where the client is a windows or linux
desktop and the back end is pyhton middleware (im going to write)
running on linux with a postgres SQL back end.

I started out using the typical client/server approach starting with
my data model. The more i've used python, the more i've gotten to
think I really dont need a relational data model at all.
...
I have one other question about the pickle system. If I add new
attributes to an object and try to unpickle an old version will it
work? How do you handle versioning of objects with pickle?

If you don't need a relational model, why not consider ZODB/ZEO
(http://zope.org/Wikis/ZODB/FrontPage)? It's the distributed Python object
database that underlies Zope. I've used it successfully for a similar
project. ZODB has transactions/versioning, but client-server syncing is a
trickier, application specific issue.

My last questions involve using sockets as a transport. If I use
python to exchange data via sockets on the server, will my server be
susceptible to buffer overflow attacks? Because I may need to support
handhelds with no SSL capability, I may need to expose a socket to the
internet unsecured. Any idea what the best approach would be to
keeping the bad people out in this instance? How should I secure my
middleware if I cannot support SSL?

Python's strings will protect you from buffer-overflow attacks caused by
sloppy C code reading data into fixed-length buffers, but even then you have
to account for potentially malicious clients sending, for example, megabytes
of data. If you want to limit message sizes, use self-delimiting netstrings
(http://cr.yp.to/proto/netstrings.txt).

It should be easy enough to encrypt your sessions (thus allowing for secure
authentication) by using something like AES. If you control the server and
the clients, SSL is overkill anyway.

HTH,

John

 

[Larry goodman]

If you don't need a relational model, why not consider ZODB/ZEO

(http://zope.org/Wikis/ZODB/FrontPage)? It's the distributed Python object
database that underlies Zope. I've used it successfully for a similar
project. ZODB has transactions/versioning, but client-server syncing is a
trickier, application specific issue.

Ill have a look. Sounds terrific.

It should be easy enough to encrypt your sessions (thus allowing for secure
authentication) by using something like AES. If you control the server and
the clients, SSL is overkill anyway.

Sounds great. Thanks for the suggestions.

 

[Larry goodman]

im looking at the ZODB documentation and I Ran across this:

"The Persistent base class is an ExtensionClass class. As a result, it
not compatible with new-style classes or types in Python 2.2 and up."

Does this mean ZODB wont work with python 2.3?

 

[Doveclaw]

Too my knowledge Python 2.3 did not remove "classic" classes, so it
obviously should work. If your using 2.2 and it works, that would be a
second way to answer your question .

 

[Sol Rosenberg]

No, it means if you want to subclass Persistent, you'll have to make it an

old-style class in ZODBs released to date. This will change in ZODB 3.3
(not yet released).

Sorry but what do you mean by "old-style" class? im new to python.
If i have an other ZODB related questions, ill post to the ZODB list.

thx

 

[John E. Barham]

im looking at the ZODB documentation and I Ran across this:

"The Persistent base class is an ExtensionClass class. As a result, it
not compatible with new-style classes or types in Python 2.2 and up."

Does this mean ZODB wont work with python 2.3?

No, it means the documentation is out of date.

The release notes for ZODB 3.2 at http://www.zope.org/Products/ZODB3.2 say
that it requires 2.2 or up. I downloaded the Windows version for 2.3 and
got it working without a hitch.

 

[Shalabh Chaturvedi]

I have one other question about the pickle system. If I add new

attributes to an object and try to unpickle an old version will it
work? How do you handle versioning of objects with pickle?

Your unpickle will succeed but any new attributes will not be present.
The object will get unpickled with exactly the same attributes it had
when it was pickled. Look at __getstate__() and __setstate__() in the
Python docs - you might be able to fix the unpickled object by
overriding __setstate__().

So I came up with the idea of just
storing all of the python objects in the postgres database pickled in
a blob field. Im sure about ten thousand other people came up with
this idea before me. The only other data stored in the table will be
the id of the object and the last time the object was modified.

What kind of attributes do your objects have? If they point to each
other, or to common objects, then pickling and unpickling can lead to
all sorts of undesirable effects, and you might be better of using a
different solution.

You might want to look at the following (some of which I have never
looked at myself):

1. ZODB (object database of http://www.zope.org/ )
2. MiddleKit in Webware (http://webware.sourceforge.net/ )
3. Twisted (http://www.twistedmatrix.com/ )
4. Metakit (http://equi4.com/metakit.html )

 

[Larry goodman]

What kind of attributes do your objects have? If they point to each

other, or to common objects, then pickling and unpickling can lead to
all sorts of undesirable effects, and you might be better of using a
different solution.

Yeh im going to have a lot of pointing going on and it confuses me how
I am going to pickle this all. I guess i'll have to store the object
id instead of a reference in the pickle and restore the reference
myself?

You might want to look at the following (some of which I have never
looked at myself):

1. ZODB (object database of http://www.zope.org/ )

This is pretty close to what I need to do.

thx