Serializing functions

[Matteo Landi]

Some weeks ago, here on the mailing list I read about picloud[1], a
python library used for cloud-computing; I was impressed by its
simplicity, here is an example:
100

So, I tried to figure out how to achieve the same result, i.e. define a
local generic function and send it somewhere for remote execution, and
the get the result back.
So I thought about serialization (pickle): I made a few tries and it
seemed to work.. but I was wrong, because I never tried to load pickled
data from another session different from the one used to pickle data
itself. If you try and pickle a function, it is not pickled as a whole,
indeed, once you unpickle it, it will raise an exception telling you
that the target function was not found in the current module.

So I'm here, with nothing in my hands; how would you implement this?

Thanks in advance.

[1] http://www.picloud.com/

 

[Bruno Desthuilliers]

Matteo Landi a écrit :

Some weeks ago, here on the mailing list I read about picloud[1], a
python library used for cloud-computing; I was impressed by its
simplicity, here is an example:
100

So, I tried to figure out how to achieve the same result, i.e. define a
local generic function and send it somewhere for remote execution, and
the get the result back.
So I thought about serialization (pickle): I made a few tries and it
seemed to work.. but I was wrong, because I never tried to load pickled
data from another session different from the one used to pickle data
itself. If you try and pickle a function, it is not pickled as a whole,
indeed, once you unpickle it, it will raise an exception telling you
that the target function was not found in the current module.

So I'm here, with nothing in my hands; how would you implement this?

Hint found on picloud's website:

"""
PiCloud's programming library, cloud, automatically transfers the
necessary source code, byte code, execution state, and data to run your
function on our cluster.
"""

 

[Paul Rubin]

If you try and pickle a function, it is not pickled as a whole,
indeed, once you unpickle it, it will raise an exception telling you
that the target function was not found in the current module.

So I'm here, with nothing in my hands; how would you implement this?

Use marshal rather than pickle. Note that requires both ends to be
running the same python version. Be aware of the obvious security
issues of running code that came from remote machine, either because the
remote machine itself may be untrusted or compromised, or because the
network between the two machines may be compromised (remote machine is
being spoofed or commnications are being modified).

 

[Matteo Landi]

Use marshal rather than pickle. Note that requires both ends to be

I could be wrong, but it seems functions are not marshable objects, is
it right?

 

[Paul Rubin]

I could be wrong, but it seems functions are not marshable objects, is
it right?

Hmm, you're right, you can marshal code objects, but you can't marshal a
function directly. It's been a while since I've used marshal and I
forgot how it works. You might be able to concoct something around it,
but it could be messy. It may be simplest to send the other side a
python source file that it can import.

 

[Matteo Landi]

Use marshal rather than pickle. Note that requires both ends to be

I could be wrong, but it seems functions are not marshable objects, is
it right?

 

[Andreas Löscher]

Am Donnerstag, den 17.06.2010, 08:18 -0700 schrieb Paul Rubin:

Hmm, you're right, you can marshal code objects, but you can't marshal a
function directly. It's been a while since I've used marshal and I
forgot how it works. You might be able to concoct something around it,
but it could be messy. It may be simplest to send the other side a
python source file that it can import.

There was a similar thread a while ago.

You can marshal the functions code object and create a new function on
the remote side using this code object.

now marshal the code object

This works as long as you are using byte code compatible Versions on
both side. The creation of functions has also some optional arguments
you must be aware of, in order to get the result you want.

Look at help(types.FunctionType) for further information.

 

[Andreas Löscher]

Am Donnerstag, den 17.06.2010, 18:03 +0200 schrieb Andreas Löscher:

Am Donnerstag, den 17.06.2010, 08:18 -0700 schrieb Paul Rubin:

There was a similar thread a while ago.

Ah I found it:
Search for:

"Object serialization: transfer from a to b (non-implemented code on b)"

in this mailing list.

Is there a good way to link a former thread?

 

[Chris Rebert]

Am Donnerstag, den 17.06.2010, 18:03 +0200 schrieb Andreas Löscher:

Ah I found it:
Search for:

"Object serialization: transfer from a to b (non-implemented code on b)"

 in this mailing list.

Is there a good way to link a former thread?

Link to the python.org archive of python-list or the google groups
c.l.p thread. In this case:
http://groups.google.com/group/comp...read/thread/6c65564f644e327f/62ef889a6e8ab886

Cheers,
Chris