Object serialization: transfer from a to b (non-implemented code on b)

G

Gabriel Rossetti

Hello everyone,

I am trying to serialize a function, class, etc and transfer it, have it
unserialized and used. The thing is that this code is not defined on the
receiving side and thus it does not work. I tried the following tests:

Terminal A:

Terminal B:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/usr/lib/python2.5/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.5/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python2.5/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'test'


Ok, so pickle needs the code to be defined on both sides, so I tried
marshal:

Terminal A:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

Ok, as the docs say, marshal is quite limited, no functions or user
classes can be marshalled, I did get it to work like this though :
>>> import inspect
>>> marshal.dumps(inspect.getmembers(test, inspect.iscode)[0][1])
'c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'

ok, but can I unmarshal it?

Terminal B:
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')
<code object test at 0xb7591578, file "<stdin>", line 1>

ok, it seams to work...
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'code' object is not callable

ok, logical, it's a code object...
eval(marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'))

ok, this works, not super pretty, but it works

now user objects?
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

ok, expected...
[]

ok, not cool, can't use the previous method...


Ok, now a word about what I'm trying to do, I'm trying to run python
code in a new process, like multiprocessing does, but without
multiprocessing. I'm trying to serialise the code because I 'd like to
do something like:

pseudo-code, serialize/unserialize/process are made up:

def newProcess(func, *args, **kwargs):
func = serialize(func)
args = serialize(args)
kwargs = serialize(kwargs)
process("python -c 'import serialize, unserialize;print
serialize(unserialize(\'%s\')(*unserialize(\'%s\'),
**unserialize(\'%s\')))'" % (func, args, kwargs))
#read result from stdout

I'm greatly simplifying this because I'm using a framework, but that's
the basic idea.

thanks,
Gabriel
 
P

Paul Rubin

Gabriel Rossetti said:
I am trying to serialize a function, class, etc and transfer it

You mean the actual code? You have to use marshal rather than pickle,
the Python versions have to be the same at both ends, and you better
have some kind of authenticated transport to stop malicious code from
getting accepted and run by the receiving end.
 
G

Gabriel Rossetti

Paul said:
You mean the actual code? You have to use marshal rather than pickle,
the Python versions have to be the same at both ends, and you better
have some kind of authenticated transport to stop malicious code from
getting accepted and run by the receiving end.

Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
my examples
 
A

Andreas Löscher

Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
my examples

The marshal module can be used to dump the Code of an Function (or
anything else), like you have done. If you have the Code, you can
reconstruct the function.
<function a at 0x7f6308a66de8>

The second parameter is the global scoop of the function.

If you want to marshal a class, things get a little more complicated,
because these consist of many code objects and you also need the code
to assemble them together.

There are two easy ways I can think of right now:

1.
Send the string of the code you want to transfer:
and then:
2.
Encapsule the stuff you want to send in a function.


Best,
Andreas
 
G

Gabriel Rossetti

Andreas said:
Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:


The marshal module can be used to dump the Code of an Function (or
anything else), like you have done. If you have the Code, you can
reconstruct the function.


...

<function a at 0x7f6308a66de8>

The second parameter is the global scoop of the function.

If you want to marshal a class, things get a little more complicated,
because these consist of many code objects and you also need the code
to assemble them together.

There are two easy ways I can think of right now:

1.
Send the string of the code you want to transfer:


and then:


2.
Encapsule the stuff you want to send in a function.


Best,
Andreas
Thanks Andreas, I'll give that a try. I didn't really want to do the
string method which is why I was trying to serialize.

best,
Gabriel
 
A

Andreas Löscher

Am Mittwoch, den 14.04.2010, 12:37 +0200 schrieb Gabriel Rossetti:
Thanks Andreas, I'll give that a try. I didn't really want to do the
string method which is why I was trying to serialize.

best,
Gabriel

You can compile the string befor you send it in an code object. This way
you also don't need to compile it twice if you execute ist two times.

But be aware, that the string method may be the most save method for
doing this. Python is not ByteCode compatible between it's versions.
But if I understand you right, this is not an issue for you.

Best,
Andreas
 
G

Gabriel Rossetti

Andreas said:
Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:


The marshal module can be used to dump the Code of an Function (or
anything else), like you have done. If you have the Code, you can
reconstruct the function.


...

<function a at 0x7f6308a66de8>

What version of python do you have? If I try your code above I get :
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute '__code__'
 
A

Andreas Löscher

import types
What version of python do you have? If I try your code above I get :

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute '__code__'

I used Version 2.6 for this. __code__ is just an alias for func_code.

This should allways work, unless the implementation of the function
object changes. (I have tested it under 2.4, 2.5 and 2.6, but it should
also work in further versions)

Best,
Andreas
 
G

Gabriel Rossetti

Andreas said:
I used Version 2.6 for this. __code__ is just an alias for func_code.



This should allways work, unless the implementation of the function
object changes. (I have tested it under 2.4, 2.5 and 2.6, but it should
also work in further versions)

Best,
Andreas
Yes, it works, thank you,
Gabriel
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,898
Latest member
BlairH7607

Latest Threads

Top