pickling question

[Gary Robinson]

When you define a class in a script, and then pickle instances of that class in the same script and store them to disk, you can't load that pickle in another script. At least not the straightforward way [pickle.load(file('somefile.pickle'))]. If you try it, you get an AttributeError during the unpickling operation.

There is no problem, of course, if the class is defined in a module which is imported by the pickling script. pickle.load(file('somefile.pickle')) then works.

Rather than provide specific examples here, there's a blog post from 2005 that discusses this issue in depth and presents the problem very well: http://stefaanlippens.net/pickleproblem. (I tested in Python 2.6 yesterday and the same issue persists.)

Questions:

1) Does this have to be the case, or is it a design problem with pickles that should be remedied?

2) Is there an easier way around it than moving the class definition to a separate module? The blog post I point to above suggests putting "__module__ = os.path.splitext(os.path.basename(__file__))[0]" into the class definiton, but that's not working in my testing because when I do that, the pickling operation fails. Is there something else that can be done?

This is obviously not a huge problem. Substantial classes should usually be defined in a separate module anyway. But sometimes it makes sense for a script to define a really simple, small class to hold some data, and needing to create a separate module just to contain such a class can be a little annoying.

--

Gary Robinson
CTO
Emergent Music, LLC
personal email: (e-mail address removed)
work email: (e-mail address removed)
Company: http://www.flyfi.com
Blog: http://www.garyrobinson.net

 

[Carl Banks]

When you define a class in a script, and then pickle
instances of that class in the same script and store
them to disk, you can't load that pickle in another
script. At least not the straightforward way
[pickle.load(file('somefile.pickle'))]. If you try it,
you get an AttributeError during the unpickling
operation.

It's because when you pickle from the script it's defined in the
module is called '__main__'. When import the script file as a Python
module from another script, the module is called 'script' or whatever
the filename is. It's just the way importing works.

There is no problem, of course, if the class is defined
in a module which is imported by the pickling script.
pickle.load(file('somefile.pickle')) then works.

Rather than provide specific examples here, there's a
blog post from 2005 that discusses this issue in depth
and presents the problem very
well:http://stefaanlippens.net/pickleproblem. (I tested
in Python 2.6 yesterday and the same issue persists.)

Actually that page kind of glosses over the real issue. But never
mind that.

Questions:

1) Does this have to be the case, or is it a design
problem with pickles that should be remedied?

It won't be remedied for the just sake of pickling. However, I'm
starting to think the underlying issue should be. It's by far the
most confusing aspect of Python.

2) Is there an easier way around it than moving the
class definition to a separate module? The blog post I
point to above suggests putting "__module__ =
os.path.splitext(os.path.basename(__file__))[0]" into
the class definiton, but that's not working in my
testing because when I do that, the pickling operation
fails. Is there something else that can be done?

This is obviously not a huge problem. Substantial
classes should usually be defined in a separate module
anyway. But sometimes it makes sense for a script to
define a really simple, small class to hold some data,
and needing to create a separate module just to contain
such a class can be a little annoying.

My general recommendation is not to mix scripts a modules unless you
an expert on Python importing. A file should be either a script or a
module, not both.

Warning: It's really hard to be expert on Python importing. I've seen
confusion about how imports work even from some very smart people on
this list (most notably, from people who mistakenly believe that PEP
328 is a fix for your issue). Python importing is the biggest wart in
the language by far.

However, if you insist, I will tell you a workaround you can use. At
the very top of your script put the following lines (replacing script
with the module's filename minus the .py):

import sys
if __name__ == '__main__':
ismain = True
__name__ = 'script'
sys.modules['script'] = sys.modules['__main__']
else:
ismain = False


Then use "if ismain" instead of "if __name__ == '__main__'" to
determine if you are running from a script. This has to come before
any class definitions or it won't work. This effectively renames the
script's __main__ module to what it would have been if it had been
imported. Then proceed with pickling as usual.


Carl Banks