problem doing unpickle in an exec statement

D

Danny Shevitz

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
pass


def main():
# instantiate
t = Tree()

# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()

# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp)
fp.close()
print "unpickling directly works just fine, result = ", result

if __name__=='__main__':
main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()


# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp)
fp.close()
print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

thanks,
Danny
 
C

castironpi

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
        pass

def main():
        # instantiate  
        t = Tree()

        # create the sweet pickle
        fp = open(file_name, 'wb')
        cPickle.dump(t, fp)
        fp.close()

        # try to unpickle directly
        fp = open(file_name, 'rb')
        result = cPickle.load(fp)
        fp.close()
        print "unpickling directly works just fine, result = ", result

if __name__=='__main__':
        main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
  print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
        clean_dict = {}
        exec code_text in clean_dict
        return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()

# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
  print "processing file_name: ", file_name
  import cPickle
  fp = open(file_name, "rb")
  result = cPickle.load(fp)
  fp.close()
  print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

thanks,
Danny

Hi,

It works if you paste

# define a class
class Tree(object):
pass

into exec_pickle.py. There are inherent dilemmas in pickling
instances of non-primitive types, such as definition location, and
change in definition across versions. How does 'class Tree' fit in to
your design module?
 
P

Peter Otten

Danny said:
Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.py) to create the pickle.

create_pickle.py: (run this first)

#############################################
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# define a class
class Tree(object):
pass


def main():
# instantiate
t = Tree()

# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()

# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp)
fp.close()
print "unpickling directly works just fine, result = ", result

if __name__=='__main__':
main()
#############################################

run this second:

exec_pickle.py
#############################################
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test1.pickle'

# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''

# a function for creating functions
def create_fun(code_text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code_text)

# this works
include_fun()


# now try to load the pickle in an exec statement
code_text = '''
def include(file_name):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp)
fp.close()
print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code_text)

# run it
include_fun(file_name)

#############################################

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

Pickling saves the name of the module and the class (and of course the
instance data). Because you put the class in your main script the module
name is __main__, and when you unpickle later pickle imports __main__ which
unfortunately is now a different script that doesn't contain the/a Tree
class. To solve that problem put the Tree class in a separate module, say
tree.py, that can be imported by both create_pickle.py and exec_pickle.py.
You only need an explicit

from tree import Tree

in create_pickle.

Peter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top