writing pickle function

Discussion in 'Python' started by perfreem@gmail.com, Jan 23, 2009.

  1. Guest

    hello,

    i am using nested defaultdict from collections and i would like to
    write it as a pickle object to a file. when i try:

    from collections import defaultdict
    x = defaultdict(lambda: defaultdict(list))

    and then try to write to a pickle file, it says:

    TypeError: can't pickle function objects

    is there a way around this? it's simply a dictionary that i want to
    write to file.. this works no problems with ordinary dicts.

    thank you.
     
    , Jan 23, 2009
    #1
    1. Advertising

  2. Chris Rebert Guest

    On Fri, Jan 23, 2009 at 6:48 AM, <> wrote:
    > hello,
    >
    > i am using nested defaultdict from collections and i would like to
    > write it as a pickle object to a file. when i try:
    >
    > from collections import defaultdict
    > x = defaultdict(lambda: defaultdict(list))
    >
    > and then try to write to a pickle file, it says:
    >
    > TypeError: can't pickle function objects
    >
    > is there a way around this? it's simply a dictionary that i want to
    > write to file.. this works no problems with ordinary dicts.


    Functions aren't pickleable because they're Python code (which is
    itself not pickleable). defaultdicts contain a reference to a function
    (in your case, a function defined using lambda), which they use to
    create default values. Thus, this causes defaultdicts to not be
    pickleable.

    This is easily worked around by pickling a plain dict w/ the contents
    of the defaultdict (i.e. dict(x) ) and then doing:

    x = defaultdict(lambda: defaultdict(list)) #create empty defaultdict
    x.update(pickle.load(the_file)) #shove contents of pickled dict into
    the defaultdict

    Cheers,
    Chris

    --
    Follow the path of the Iguana...
    http://rebertia.com
     
    Chris Rebert, Jan 23, 2009
    #2
    1. Advertising

  3. On Jan 23, 2:48 pm, wrote:
    > hello,
    >
    > i am using nested defaultdict from collections and i would like to
    > write it as a pickle object to a file. when i try:
    >
    > from collections import defaultdict
    > x = defaultdict(lambda: defaultdict(list))
    >
    > and then try to write to a pickle file, it says:
    >
    > TypeError: can't pickle function objects
    >
    > is there a way around this? it's simply a dictionary that i want to
    > write to file.. this works no problems with ordinary dicts.
    >
    > thank you.



    One way via a subclass:

    (from memory)

    class MyCollection(defaultdict):

    def __init__(self):
    defaultdict.__init__(self, list)

    def __reduce__(self):
    return (MyCollection, (), None, None, self.iteritems())

    and if you are so inclined (nothing to do with pickling):

    __setattr__ = defaultdict.__setitem__
    __getattr__ = defaultdict.__getitem__

    G.
     
    Gerard Flanagan, Jan 23, 2009
    #3
  4. Peter Otten Guest

    wrote:

    > i am using nested defaultdict from collections and i would like to
    > write it as a pickle object to a file. when i try:
    >
    > from collections import defaultdict
    > x = defaultdict(lambda: defaultdict(list))
    >
    > and then try to write to a pickle file, it says:
    >
    > TypeError: can't pickle function objects
    >
    > is there a way around this? it's simply a dictionary that i want to
    > write to file.. this works no problems with ordinary dicts.


    The error message is misleading. You can pickle your defaultdict if you use
    a "normal" function instead of the lambda:

    $ cat pickle_defaultdict.py
    import sys
    from collections import defaultdict
    from cPickle import dumps, loads

    if "--lambda" in sys.argv:
    make_inner = lambda: defaultdict(list)
    else:
    def make_inner():
    return defaultdict(list)

    d = defaultdict(make_inner)
    d[1][2].append(42)
    e = loads(dumps(d))
    print e == d

    $ python pickle_defaultdict.py
    True
    $ python pickle_defaultdict.py --lambda
    Traceback (most recent call last):
    File "pickle_defaultdict.py", line 13, in <module>
    e = loads(dumps(d))
    File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle function objects

    Peter
     
    Peter Otten, Jan 23, 2009
    #4
  5. Steve Holden Guest

    Peter Otten wrote:
    > wrote:
    >
    >> i am using nested defaultdict from collections and i would like to
    >> write it as a pickle object to a file. when i try:
    >>
    >> from collections import defaultdict
    >> x = defaultdict(lambda: defaultdict(list))
    >>
    >> and then try to write to a pickle file, it says:
    >>
    >> TypeError: can't pickle function objects
    >>
    >> is there a way around this? it's simply a dictionary that i want to
    >> write to file.. this works no problems with ordinary dicts.

    >
    > The error message is misleading. You can pickle your defaultdict if you use
    > a "normal" function instead of the lambda:
    >
    > $ cat pickle_defaultdict.py
    > import sys
    > from collections import defaultdict
    > from cPickle import dumps, loads
    >
    > if "--lambda" in sys.argv:
    > make_inner = lambda: defaultdict(list)
    > else:
    > def make_inner():
    > return defaultdict(list)
    >
    > d = defaultdict(make_inner)
    > d[1][2].append(42)
    > e = loads(dumps(d))
    > print e == d
    >
    > $ python pickle_defaultdict.py
    > True
    > $ python pickle_defaultdict.py --lambda
    > Traceback (most recent call last):
    > File "pickle_defaultdict.py", line 13, in <module>
    > e = loads(dumps(d))
    > File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
    > raise TypeError, "can't pickle %s objects" % base.__name__
    > TypeError: can't pickle function objects
    >

    But can you unpickle the objects you pickle? I believe that functions
    are references relative to the modules they come from, and so won't be
    found unless the same module can be imported at unpickle time. Could be
    wrong, haven't read the source, but I understand that functions work the
    same as classes.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Jan 23, 2009
    #5
  6. Peter Otten Guest

    Steve Holden wrote:

    > Peter Otten wrote:
    >> wrote:
    >>
    >>> i am using nested defaultdict from collections and i would like to
    >>> write it as a pickle object to a file. when i try:
    >>>
    >>> from collections import defaultdict
    >>> x = defaultdict(lambda: defaultdict(list))
    >>>
    >>> and then try to write to a pickle file, it says:
    >>>
    >>> TypeError: can't pickle function objects
    >>>
    >>> is there a way around this? it's simply a dictionary that i want to
    >>> write to file.. this works no problems with ordinary dicts.

    >>
    >> The error message is misleading. You can pickle your defaultdict if you
    >> use a "normal" function instead of the lambda:
    >>
    >> $ cat pickle_defaultdict.py
    >> import sys
    >> from collections import defaultdict
    >> from cPickle import dumps, loads
    >>
    >> if "--lambda" in sys.argv:
    >> make_inner = lambda: defaultdict(list)
    >> else:
    >> def make_inner():
    >> return defaultdict(list)
    >>
    >> d = defaultdict(make_inner)
    >> d[1][2].append(42)
    >> e = loads(dumps(d))
    >> print e == d
    >>
    >> $ python pickle_defaultdict.py
    >> True
    >> $ python pickle_defaultdict.py --lambda
    >> Traceback (most recent call last):
    >> File "pickle_defaultdict.py", line 13, in <module>
    >> e = loads(dumps(d))
    >> File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
    >> raise TypeError, "can't pickle %s objects" % base.__name__
    >> TypeError: can't pickle function objects
    >>

    > But can you unpickle the objects you pickle? I believe that functions
    > are references relative to the modules they come from, and so won't be
    > found unless the same module can be imported at unpickle time. Could be
    > wrong, haven't read the source, but I understand that functions work the
    > same as classes.


    Indeed, what is stored in the pickle is the module and function name, not
    the byte code:

    >>> from cPickle import dumps, loads
    >>> def f(): print "original"

    ....
    >>> s = dumps(f)
    >>> def f(): print "updated"

    ....
    >>> g = loads(s)
    >>> g()

    updated

    That limits pickle as an exchange format to installations that provide
    compatible versions of the pickled classes and functions.

    Do you think that would be a problem for the OP?

    Peter
     
    Peter Otten, Jan 23, 2009
    #6
  7. Steve Holden Guest

    Peter Otten wrote:
    > Steve Holden wrote:
    >
    >> Peter Otten wrote:
    >>> wrote:
    >>>
    >>>> i am using nested defaultdict from collections and i would like to
    >>>> write it as a pickle object to a file. when i try:
    >>>>
    >>>> from collections import defaultdict
    >>>> x = defaultdict(lambda: defaultdict(list))
    >>>>
    >>>> and then try to write to a pickle file, it says:
    >>>>
    >>>> TypeError: can't pickle function objects
    >>>>
    >>>> is there a way around this? it's simply a dictionary that i want to
    >>>> write to file.. this works no problems with ordinary dicts.
    >>> The error message is misleading. You can pickle your defaultdict if you
    >>> use a "normal" function instead of the lambda:
    >>>
    >>> $ cat pickle_defaultdict.py
    >>> import sys
    >>> from collections import defaultdict
    >>> from cPickle import dumps, loads
    >>>
    >>> if "--lambda" in sys.argv:
    >>> make_inner = lambda: defaultdict(list)
    >>> else:
    >>> def make_inner():
    >>> return defaultdict(list)
    >>>
    >>> d = defaultdict(make_inner)
    >>> d[1][2].append(42)
    >>> e = loads(dumps(d))
    >>> print e == d
    >>>
    >>> $ python pickle_defaultdict.py
    >>> True
    >>> $ python pickle_defaultdict.py --lambda
    >>> Traceback (most recent call last):
    >>> File "pickle_defaultdict.py", line 13, in <module>
    >>> e = loads(dumps(d))
    >>> File "/usr/lib/python2.5/copy_reg.py", line 69, in _reduce_ex
    >>> raise TypeError, "can't pickle %s objects" % base.__name__
    >>> TypeError: can't pickle function objects
    >>>

    >> But can you unpickle the objects you pickle? I believe that functions
    >> are references relative to the modules they come from, and so won't be
    >> found unless the same module can be imported at unpickle time. Could be
    >> wrong, haven't read the source, but I understand that functions work the
    >> same as classes.

    >
    > Indeed, what is stored in the pickle is the module and function name, not
    > the byte code:
    >
    >>>> from cPickle import dumps, loads
    >>>> def f(): print "original"

    > ...
    >>>> s = dumps(f)
    >>>> def f(): print "updated"

    > ...
    >>>> g = loads(s)
    >>>> g()

    > updated
    >
    > That limits pickle as an exchange format to installations that provide
    > compatible versions of the pickled classes and functions.
    >
    > Do you think that would be a problem for the OP?
    >

    Nice demonstration!

    Possibly not, though the original use of lambdas demonstrated at least
    confusion. But there needs to be an understanding that the pickled
    function has to be importable. Just using a function in __main__ and
    then trying to unpickle from another program that doesn't contain the
    function won;t hack it.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Jan 24, 2009
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ted kelly

    pickle function reference ??

    ted kelly, Nov 12, 2003, in forum: Python
    Replies:
    1
    Views:
    382
    Martin v. =?iso-8859-15?q?L=F6wis?=
    Nov 12, 2003
  2. Michael Hohn
    Replies:
    3
    Views:
    1,224
    Dima Dorfman
    Oct 31, 2004
  3. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    408
  4. Michele Simionato
    Replies:
    2
    Views:
    1,937
    Michele Simionato
    May 23, 2008
  5. Terry
    Replies:
    1
    Views:
    753
    Terry
    Aug 11, 2009
Loading...

Share This Page