pickling extension class

[harold fellermann]

Hi all,

I have a problem pickling an extension class. As written in the
Extending/Embedding Manual, I
provided a function __reduce__ that returns the appropreate tuple. This
seams to work fine,
but I still cannot pickle because of the following error:
Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/sw/lib/python2.4/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
File "/sw/lib/python2.4/pickle.py", line 231, in dump
self.save(obj)
File "/sw/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/sw/lib/python2.4/pickle.py", line 414, in save_reduce
save(func)
File "/sw/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount']<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?

Any help appreceated,

- harold -

 

[Alex Martelli]

File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount']<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?

These symptomps are pretty weird -- let's try to pin things down a bit
more. The relevant few lines of pickle.py are:

try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(

so, could you please edit your pickle.py to provide VASTLY more info,
say:

try:
print 'Here it goes...:'
_xx = __import__(module)
print ' __import__ says: %r' % (_xx,)
mod = sys.modules[module]
print ' in sys.modules: %r' % (mod,)
klass = getattr(mod, name)
print ' klass is: %r' % (klass,)
except (ImportError, KeyError, AttributeError), _xx:
print ' OOPS, error (%s): %s' % (_xx.__class__, _xx)
raise PicklingError(

and let us know exactly what his modified pickle.py outputs...?


Thanks,

Alex

 

[harold fellermann]

File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(

dir(hyper)

['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount']

hyper.PeriodicGrid

<type 'hyper.PeriodicGrid'>

So pickle complains about the class PeriodicGrid not being found in
the
module hyper, but a dir()
proves that python can find it. Has anyone an idea what's going wrong
here?

These symptomps are pretty weird -- let's try to pin things down a bit
more. The relevant few lines of pickle.py are:

try:
__import__(module)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
raise PicklingError(

so, could you please edit your pickle.py to provide VASTLY more info,
[...]

and let us know exactly what his modified pickle.py outputs...?

Here it goes...:
OOPS, error (exceptions.ImportError): No module named hyper
Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 1387,
in dump
Pickler(file, protocol, bin).dump(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 231,
in dump
self.save(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 338,
in save
self.save_reduce(obj=obj, *rv)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 414,
in save_reduce
save(func)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 293,
in save
f(self, obj) # Call unbound method with explicit self
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 765,
in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <type 'hyper.PeriodicGrid'>: it's
not found as hyper.PeriodicGrid


I have noticed that the error does not occur, when the imported module
('hyper') is in the same directory as the script that pickles. When it
is imported from a subpackage (like in the code
I sent you) it goes wrong.

- harold -

 

[Alex Martelli]

Here it goes...:
OOPS, error (exceptions.ImportError): No module named hyper

So, the __import__ in pickle fails -- indeed, __import__('foo') when
'foo' ``is imported from a subpackage'' is _supposed_ to fail, as
'hyper' is not the name you SHOULD be importing. For example,

__import__('email.Encoders')

is fine, but just

__import__('Encoders')

fails with ImportError -- there IS no toplevel module by that name!

I have noticed that the error does not occur, when the imported module
('hyper') is in the same directory as the script that pickles. When it
is imported from a subpackage (like in the code
I sent you) it goes wrong.

If you can't fix the modulename given by your type, you can perhaps
kludge things up by (e.g.)

import the.real.hyper
sys.modules['hyper']=the.real.hyper

before you pickle; but I suspect UNpickling would fail in that case.

Using the standard library copy_reg module to register your way to
pickle and recover instances of your type might work better.


Alex

 

[David M. Cooke]

Hi all,

I have a problem pickling an extension class. As written in the
Extending/Embedding Manual, I
provided a function __reduce__ that returns the appropreate tuple.
This seams to work fine,
but I still cannot pickle because of the following error:
Traceback (most recent call last):
File "pickle_test.py", line 5, in ?
pickle.dump(g,file("test","w"))
File "/sw/lib/python2.4/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
File "/sw/lib/python2.4/pickle.py", line 231, in dump
self.save(obj)
File "/sw/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/sw/lib/python2.4/pickle.py", line 414, in save_reduce
save(func)
File "/sw/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/sw/lib/python2.4/pickle.py", line 760, in save_global
raise PicklingError(
['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
'__file__', '__name__', 'refcount']<type 'hyper.PeriodicGrid'>

^^^^^

I think that's your error. The extension type is declared to be
hyper.PeriodicGrid, where it actually is model.hyper.PeriodicGrid
(because hyper is in the model package).

Pickle stores g.__class__.__module__ (which is "hyper") and
g.__class__.__name__ (="PeriodicGrid") to find the class object for
reimporting, and on unpickling, tries to do __import__("hyper"), which
fails.

The tp_name slot of your extension type should be "model.hyper.PeriodicGrid".