Best way to pickle functions

[Aaron Scott]

I have a number of functions that I need to pickle without necessarily
knowing their names in advance. My first thought was to put all the
functions in a class, then pickle the class, but it doesn't really
work like I expected it to.

import cPickle
class PickleClass:
def Awesome(self):
pass
stored = cPickle.dumps(PickleClass)
del PickleClass
restored = cPickle.loads(stored)

Results:

Traceback (most recent call last):
File "pickletest.py", line 7, in <module>
restored = cPickle.loads(stored)
AttributeError: 'module' object has no attribute 'PickleClass'

So, the class itself isn't being pickled, just an instance of it.

This being the case, what's the best way to store these functions?
Maybe dump the class to a string and pull them back with an exec
statement when I need them?

 

[Aaron Brady]

I have a number of functions that I need to pickle without necessarily
knowing their names in advance. My first thought was to put all the
functions in a class, then pickle the class, but it doesn't really
work like I expected it to.

        import cPickle
        class PickleClass:
                def Awesome(self):
                        pass
        stored = cPickle.dumps(PickleClass) snip
So, the class itself isn't being pickled, just an instance of it.

This being the case, what's the best way to store these functions?
Maybe dump the class to a string and pull them back with an exec
statement when I need them?

All pickling a class does is pickle its module and name. You can't
pickle functions in principle because byte-code is sensitive and
volatile, and is least likely to run consistently later on. 'pickle'
is not just for serialization, it's for persistence.

Pickling the source code is much sturdier. It's very unlikely that
the same code runs differently in different interpreters. It's much
more likely that the same code runs the same, or not at all.

It's better yet to just get the source from the original place every
time: instead, pickle a file name and open the file.

 

[Aaron Scott]

Pickling the source code is much sturdier.  It's very unlikely that

the same code runs differently in different interpreters.  It's much
more likely that the same code runs the same, or not at all.

Okay, I've run into another problem. I've saved the code to a string,
so I can call it up when I need it. I want to keep these functions all
together, though, so I'm pushing them into a dictionary when I execute
it. It seems like when I put it in a dictionary, though, it messes up
the scope of the functions contained within. For example:

import cPickle
def Main():
holder = {}
functiontest = "def PickleTest():\n\tprint cPickle"
exec functiontest in holder
print holder["PickleTest"]()
Main()

.... produces:

Traceback (most recent call last):
File "pickletest.py", line 11, in <module>
Main()
File "pickletest.py", line 9, in Main
print holder["PickleTest"]()
File "<string>", line 2, in PickleTest
NameError: global name 'cPickle' is not defined

Is there any way to do this so that the functions have access to the
higher scope?

Thanks.

 

[Aaron Scott]

Never mind. Solved the problem by putting the functions in a class and
dumping that into a string. Then, when I need it, I executed the
string to get myself the class, then created an instance of that class
which gave me access to those functions along with the correct scope.
Probably not the smartest solution, but it works for now.

 

[Dave Angel]

Pickling the source code is much sturdier. It's very unlikely that
the same code runs differently in different interpreters. It's much
more likely that the same code runs the same, or not at all.

Okay, I've run into another problem. I've saved the code to a string,
so I can call it up when I need it. I want to keep these functions all
together, though, so I'm pushing them into a dictionary when I execute
it. It seems like when I put it in a dictionary, though, it messes up
the scope of the functions contained within. For example:

import cPickle
def Main():
holder =}
functiontest =def PickleTest():\n\tprint cPickle"
exec functiontest in holder
print holder["PickleTest"]()
Main()

... produces:

Traceback (most recent call last):
File "pickletest.py", line 11, in <module>
Main()
File "pickletest.py", line 9, in Main
print holder["PickleTest"]()
File "<string>", line 2, in PickleTest
NameError: global name 'cPickle' is not defined

Is there any way to do this so that the functions have access to the
higher scope?

Thanks.

Why not use import ? Simply recreate the source file, if necessary, and
import it again.

If you must import without it being in a clear text file, check out the
deprecated imputil module, standard in 2.x, but removed in Python 3.0.
And if you find some way to get __import__ or one of its relatives to
work from a stream instead of a file, please let us know.

 

[Aaron Scott]

Why not use import ?  Simply recreate the source file, if necessary, and

import it again.

Ah, you'd think it would be that easy

The problem with just importing a module is that the module is then
cached in memory. Multiple copies of the program are running on a
server, and each of them have something akin to a "randomfunctions"
module. When the first program is accessed, it loads
"randomfunctions". When the second program is accessed, it uses the
"randomfunctions" module already in memory, even though it doesn't
contain the right functions. So, I have to pull in these functions
dynamically.

 

[Aaron Brady]

Ah, you'd think it would be that easy

The problem with just importing a module is that the module is then
cached in memory. Multiple copies of the program are running on a
server, and each of them have something akin to a "randomfunctions"
module. When the first program is accessed, it loads
"randomfunctions". When the second program is accessed, it uses the
"randomfunctions" module already in memory, even though it doesn't
contain the right functions. So, I have to pull in these functions
dynamically.

Here I found this cookie you can have it. </cookie>

imp.load_source(name, pathname[, file])
Load and initialize a module implemented as a Python source file and
return its module object. If the module was already initialized, it
will be initialized again.

'again' was actually in italics in the docs. </much rejoicing>

I presume you're aware of eval and exec.

 

[azrael]

As in Python everythong is an object you could use __name__.
def2

in this case you can combine __name__ to get the object name and then
combine it with eval to pickle.

in this way you deal with dynamic strings and you can pickle anything