writing pickle function

[perfreem]

hello,

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

thank you.

 

[Chris Rebert]

hello,

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

Functions aren't pickleable because they're Python code (which is
itself not pickleable). defaultdicts contain a reference to a function
(in your case, a function defined using lambda), which they use to
create default values. Thus, this causes defaultdicts to not be
pickleable.

This is easily worked around by pickling a plain dict w/ the contents
of the defaultdict (i.e. dict(x) ) and then doing:

x = defaultdict(lambda: defaultdict(list)) #create empty defaultdict
x.update(pickle.load(the_file)) #shove contents of pickled dict into
the defaultdict

Cheers,
Chris

 

[Gerard Flanagan]

hello,

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

thank you.

One way via a subclass:

(from memory)

class MyCollection(defaultdict):

def __init__(self):
defaultdict.__init__(self, list)

def __reduce__(self):
return (MyCollection, (), None, None, self.iteritems())

and if you are so inclined (nothing to do with pickling):

__setattr__ = defaultdict.__setitem__
__getattr__ = defaultdict.__getitem__

G.

 

[Peter Otten]

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

The error message is misleading. You can pickle your defaultdict if you use
a "normal" function instead of the lambda:

$ cat pickle_defaultdict.py
import sys
from collections import defaultdict
from cPickle import dumps, loads

if "--lambda" in sys.argv:
make_inner = lambda: defaultdict(list)
else:
def make_inner():
return defaultdict(list)

d = defaultdict(make_inner)
d[1][2].append(42)
e = loads(dumps(d))
print e == d

$ python pickle_defaultdict.py
True
$ python pickle_defaultdict.py --lambda
Traceback (most recent call last):
File "pickle_defaultdict.py", line 13, in <module>
e = loads(dumps(d))
File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Peter

 

[Steve Holden]

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

The error message is misleading. You can pickle your defaultdict if you use
a "normal" function instead of the lambda:

$ cat pickle_defaultdict.py
import sys
from collections import defaultdict
from cPickle import dumps, loads

if "--lambda" in sys.argv:
make_inner = lambda: defaultdict(list)
else:
def make_inner():
return defaultdict(list)

d = defaultdict(make_inner)
d[1][2].append(42)
e = loads(dumps(d))
print e == d

$ python pickle_defaultdict.py
True
$ python pickle_defaultdict.py --lambda
Traceback (most recent call last):
File "pickle_defaultdict.py", line 13, in <module>
e = loads(dumps(d))
File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

But can you unpickle the objects you pickle? I believe that functions
are references relative to the modules they come from, and so won't be
found unless the same module can be imported at unpickle time. Could be
wrong, haven't read the source, but I understand that functions work the
same as classes.

regards
Steve

 

[Peter Otten]

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.

The error message is misleading. You can pickle your defaultdict if you
use a "normal" function instead of the lambda:

$ cat pickle_defaultdict.py
import sys
from collections import defaultdict
from cPickle import dumps, loads

if "--lambda" in sys.argv:
make_inner = lambda: defaultdict(list)
else:
def make_inner():
return defaultdict(list)

d = defaultdict(make_inner)
d[1][2].append(42)
e = loads(dumps(d))
print e == d

$ python pickle_defaultdict.py
True
$ python pickle_defaultdict.py --lambda
Traceback (most recent call last):
File "pickle_defaultdict.py", line 13, in <module>
e = loads(dumps(d))
File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

But can you unpickle the objects you pickle? I believe that functions
are references relative to the modules they come from, and so won't be
found unless the same module can be imported at unpickle time. Could be
wrong, haven't read the source, but I understand that functions work the
same as classes.

Indeed, what is stored in the pickle is the module and function name, not
the byte code:
updated

That limits pickle as an exchange format to installations that provide
compatible versions of the pickled classes and functions.

Do you think that would be a problem for the OP?

Peter

 

[Steve Holden]

(e-mail address removed) wrote:

i am using nested defaultdict from collections and i would like to
write it as a pickle object to a file. when i try:

from collections import defaultdict
x = defaultdict(lambda: defaultdict(list))

and then try to write to a pickle file, it says:

TypeError: can't pickle function objects

is there a way around this? it's simply a dictionary that i want to
write to file.. this works no problems with ordinary dicts.
The error message is misleading. You can pickle your defaultdict if you
use a "normal" function instead of the lambda:

$ cat pickle_defaultdict.py
import sys
from collections import defaultdict
from cPickle import dumps, loads

if "--lambda" in sys.argv:
make_inner = lambda: defaultdict(list)
else:
def make_inner():
return defaultdict(list)

d = defaultdict(make_inner)
d[1][2].append(42)
e = loads(dumps(d))
print e == d

$ python pickle_defaultdict.py
True
$ python pickle_defaultdict.py --lambda
Traceback (most recent call last):
File "pickle_defaultdict.py", line 13, in <module>
e = loads(dumps(d))
File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

But can you unpickle the objects you pickle? I believe that functions
are references relative to the modules they come from, and so won't be
found unless the same module can be imported at unpickle time. Could be
wrong, haven't read the source, but I understand that functions work the
same as classes.

Indeed, what is stored in the pickle is the module and function name, not
the byte code:
updated

That limits pickle as an exchange format to installations that provide
compatible versions of the pickled classes and functions.

Do you think that would be a problem for the OP?

Nice demonstration!

Possibly not, though the original use of lambdas demonstrated at least
confusion. But there needs to be an understanding that the pickled
function has to be importable. Just using a function in __main__ and
then trying to unpickle from another program that doesn't contain the
function won;t hack it.

regards
Steve