cPickle problems

[Jeff Poole]

This is going to be a pretty vague message because it involves a large
block of code I'd rather avoid posting. Basically, I've been pickling
a dictionary of instances of a class I've created (which contains
references to other instances of other classes). At some point in the
last few weeks, pickling has stopped working with the following error:

Traceback (most recent call last):
File "./generateTools.py", line 50, in ?
proj.findVHDLfiles("vhd")
File "/cygdrive/c/fpga/tools/FPGAProject.py", line 630, in
findVHDLfiles
cpickle.dump(filenameToClassDic,pickleFile,PICKLE_FILE_PROTOCOL)
File "/usr/lib/python2.4/copy_reg.py", line 83, in _reduce_ex
dict = getstate()
TypeError: 'str' object is not callable


Does anyone have any idea why this might be failing in this way? It's
driving me crazy. If need be I can probably provide the code to the
class, though it's about 1000 lines, most of which is PyParsing code.
I appreciate any help you guys can provide.


Jeff

 

[Jeff Poole]

Oh, and I suppose I should provide some version information:

$ python
Python 2.4.3 (#1, May 18 2006, 07:40:45)
[GCC 3.3.3 (cygwin special)] on cygwin

 

[Jeff Poole]

To clutter this up with yet another message, this is what happens if I
use pickle instead of cPickle. Same error, but it has more of a stack
trace so someone more advanced than myself might be able to pick out
what is going amiss.

Traceback (most recent call last):
File "./generateTools.py", line 50, in ?
proj.findVHDLfiles("vhd")
File "/cygdrive/c/fpga/tools/FPGAProject.py", line 631, in
findVHDLfiles
cpickle.dump(filenameToClassDic,pickleFile,PICKLE_FILE_PROTOCOL)
File "/usr/lib/python2.4/pickle.py", line 1382, in dump
Pickler(file, protocol, bin).dump(obj)
File "/usr/lib/python2.4/pickle.py", line 231, in dump
self.save(obj)
File "/usr/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.4/pickle.py", line 663, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.4/pickle.py", line 677, in _batch_setitems
save(v)
File "/usr/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.4/pickle.py", line 433, in save_reduce
save(state)
File "/usr/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.4/pickle.py", line 663, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.4/pickle.py", line 677, in _batch_setitems
save(v)
File "/usr/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.4/pickle.py", line 614, in save_list
self._batch_appends(iter(obj))
File "/usr/lib/python2.4/pickle.py", line 629, in _batch_appends
save(x)
File "/usr/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.4/pickle.py", line 576, in save_tuple
save(element)
File "/usr/lib/python2.4/pickle.py", line 338, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.4/pickle.py", line 433, in save_reduce
save(state)
File "/usr/lib/python2.4/pickle.py", line 293, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.4/pickle.py", line 663, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.4/pickle.py", line 677, in _batch_setitems
save(v)
File "/usr/lib/python2.4/pickle.py", line 313, in save
rv = reduce(self.proto)
File "/usr/lib/python2.4/copy_reg.py", line 83, in _reduce_ex
dict = getstate()
TypeError: 'str' object is not callable

Oh, and I suppose I should provide some version information:

$ python
Python 2.4.3 (#1, May 18 2006, 07:40:45)
[GCC 3.3.3 (cygwin special)] on cygwin

This is going to be a pretty vague message because it involves a large
block of code I'd rather avoid posting. Basically, I've been pickling
a dictionary of instances of a class I've created (which contains
references to other instances of other classes). At some point in the
last few weeks, pickling has stopped working with the following error:

Traceback (most recent call last):
File "./generateTools.py", line 50, in ?
proj.findVHDLfiles("vhd")
File "/cygdrive/c/fpga/tools/FPGAProject.py", line 630, in
findVHDLfiles
cpickle.dump(filenameToClassDic,pickleFile,PICKLE_FILE_PROTOCOL)
File "/usr/lib/python2.4/copy_reg.py", line 83, in _reduce_ex
dict = getstate()
TypeError: 'str' object is not callable


Does anyone have any idea why this might be failing in this way? It's
driving me crazy. If need be I can probably provide the code to the
class, though it's about 1000 lines, most of which is PyParsing code.
I appreciate any help you guys can provide.


Jeff

 

[Gabriel Genellina]

File "/usr/lib/python2.4/pickle.py", line 313, in save
rv = reduce(self.proto)
File "/usr/lib/python2.4/copy_reg.py", line 83, in _reduce_ex
dict = getstate()
TypeError: 'str' object is not callable

Someway, self.__getstate__ is a string, not a method...
Since it fails inside a python module, you could print some debug
information to see what happens, like repr(self), type(self), repr(getstate)...



--
Gabriel Genellina
Softlab SRL

__________________________________________________
Correo Yahoo!
Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar

 

[Jeff Poole]

Good idea. Well, I did that, and I found out that the object causing
problems is a ParseResults object (a class from PyParsing) and that the
__getstate__ member is in fact an empty string (''). I'm not sure
where this leaves me... The PyParsing code clearly never creates such
a member and my code never creates it. In fact, searching all the code
involved (not including what is in /usr/lib/python2.4 ), shows no use
of __getstate__ at all (or even the string "getstate").

Ok, I figured it out. ParseResults has the following member:

def __getattr__( self, name ):
if name not in self.__slots__:
if self.__tokdict.has_key( name ):
if name not in self.__accumNames:
return self.__tokdict[name][-1][0]
else:
return ParseResults([ v[0] for v in
self.__tokdict[name] ])
else:
return ""
return None

So when something tries to retrieve .__getstate__, it returns an empty
string. If I tell it to raise an AttributeException instead, then I
get this message:

TypeError: a class that defines __slots__ without defining __getstate__
cannot be pickled

I think I need to talk to the author of PyParsing about this one...
Thanks for your help!


Jeff

 

[Klaas]

Good idea. Well, I did that, and I found out that the object causing
problems is a ParseResults object (a class from PyParsing) and that the
__getstate__ member is in fact an empty string (''). I'm not sure
where this leaves me... The PyParsing code clearly never creates such

Sounds like ParseResults is not intended to be pickable.

-Mike

 

[Paul McGuire]

So when something tries to retrieve .__getstate__, it returns an empty
string. If I tell it to raise an AttributeException instead, then I
get this message:

TypeError: a class that defines __slots__ without defining __getstate__
cannot be pickled

I think I need to talk to the author of PyParsing about this one...
Thanks for your help!


Jeff

Jeff -

Sorry not to reply sooner, I have been out of town with limited access to
c.l.py. Also, I didn't really pick up on "cPickle problems" as a
pyparsing-related post.

I would be happy to make ParseResults picklable, I didn't realize that this
getattr behavior was defeating that capability. I have had people
successfully use pickle on pyparsing grammars, but have not had anyone
report any pickle experience with ParseResults objects, either pro or con.

I have a test version of pyparsing with pickling support added, and it
passes my simple unit tests. I could send it to you if you would test it
against your program.

-- Paul

 

[Jeff Poole]

Jeff -

Sorry not to reply sooner, I have been out of town with limited access to
c.l.py. Also, I didn't really pick up on "cPickle problems" as a
pyparsing-related post.

I would be happy to make ParseResults picklable, I didn't realize that this
getattr behavior was defeating that capability. I have had people
successfully use pickle on pyparsing grammars, but have not had anyone
report any pickle experience with ParseResults objects, either pro or con.

I have a test version of pyparsing with pickling support added, and it
passes my simple unit tests. I could send it to you if you would test it
against your program.

-- Paul

Paul,

No problem, I think we've both been busy with other things. I was
actually going to post something on the PyParsing wikispaces since I
wasn't sure if you checked here. I wasn't actually intentionally
pickling any ParseResults objects, but I realized that in my parse
actions, I was making a copy of the "t" parameter, which is actually a
ParseResults object. I switched my code to use t.asList() and things
are ok now. If you'd like me to test your version, I'd be glad to,
otherwise I can wait until whenever that change makes it's way into a
release.

Jeff

 

[John Machin]

Choice of keywords is important -- for example, "Excel" always grabs my
attention. In this case, the keyword the OP should have used is not
"cPickle" but "re"


[snip]

Paul,

No problem, I think we've both been busy with other things. I was
actually going to post something on the PyParsing wikispaces since I
wasn't sure if you checked here.

I suggest that the OP take his googler back to the shop he bought it
from; it is severely damaged.


HTH,
John