importing class objects from a pickled file

Discussion in 'Python' started by Catherine Moroney, May 4, 2011.

  1. Hello,

    I have an object of class X that I am writing to a pickled file. The
    pickling part goes fine, but I am having some problems reading the
    object back out, as I get complaints about "unable to import module X".

    The only way I have found around it is to run the read-file code out of
    the same directory that contains the X.py file, but this is obviously
    not a portable way of doing things.

    Even when I put statements into the code such as "from Y.X import X"
    where Y is the name of the python package that contains the X,py file,
    the import statement works, but I am still unable to read the object
    from the pickled file, running into the same "unable to import module X"
    error.

    Am I explaining myself properly? Why doesn't the code that loads the
    object from the pickled file work unless I am sitting in the same
    directory? The code that writes the pickled file has the statement
    "from Y.X import X" statement" at the top, as does the reading code, but
    even though that import statement succeeds, the read still fails with
    the import error.

    Thanks for any help,

    Catherine
    Catherine Moroney, May 4, 2011
    #1
    1. Advertising

  2. Catherine Moroney

    James Mills Guest

    On Wed, May 4, 2011 at 10:18 AM, Catherine Moroney
    <> wrote:
    > Am I explaining myself properly?  Why doesn't the code that loads the
    > object from the pickled file work unless I am sitting in the same directory?
    >   The code that writes the pickled file has the statement
    > "from Y.X import X" statement" at the top, as does the reading code, but
    > even though that import statement succeeds, the read still fails with the
    > import error.


    pickled objects won't work this way unless you "customize" the
    way in which your class gets serialized.

    See: http://docs.python.org/library/pickle.html#the-pickle-protocol

    Any time you want to unpickle a user class, that class must be available.

    I suggest serializing to a more common format (say JSON) and re-create
    your class with the data.

    cheers
    James

    --
    -- James Mills
    --
    -- "Problems are solved by method"
    James Mills, May 4, 2011
    #2
    1. Advertising

  3. On 2011-05-03 20:18:33 -0400, Catherine Moroney said:

    > Hello,
    >
    > I have an object of class X that I am writing to a pickled file. The
    > pickling part goes fine, but I am having some problems reading the
    > object back out, as I get complaints about "unable to import module X".
    >
    > The only way I have found around it is to run the read-file code out of
    > the same directory that contains the X.py file, but this is obviously
    > not a portable way of doing things.
    >
    > Even when I put statements into the code such as "from Y.X import X"
    > where Y is the name of the python package that contains the X,py file,
    > the import statement works, but I am still unable to read the object
    > from the pickled file, running into the same "unable to import module X"
    > error.
    >
    > Am I explaining myself properly? Why doesn't the code that loads the
    > object from the pickled file work unless I am sitting in the same directory?


    That's not the actual requirement.

    The pickled representation of an instance of a user-defined class contains:

    * the name of the class
    * the name of the class's module
    * the object's attributes, pickled recursively

    You can have a look at the representation using protocol zero, which is
    printable and not terribly hard to read. Here's an example, lightly
    annotated:

    >>> import pickle
    >>> class Example(object):

    ... def __init__(self, x):
    ... self.x = x
    ...
    >>> one = Example(1)
    >>> print pickle.dumps(one, 0)

    ccopy_reg
    _reconstructor
    p0
    module name (c__main__
    class name Example
    p1
    c__builtin__
    object
    p2
    Ntp3
    Rp4
    (dp5
    field name S'x'
    p6
    field value I1
    sb.
    >>>


    Obviously, since the pickled representation only contains the *name and
    location* of the class and not the class itself, you need to have the
    definition of the class handy when unpickling an object. This means
    that the module that defines the class must be on the module search
    path - either because it's in the current directory (or a package in
    the current directory, or..), or because it's in PYTHONPATH, or because
    it's somewhere else on sys.path.

    > The code that writes the pickled file has the statement
    > "from Y.X import X" statement" at the top, as does the reading code, but
    > even though that import statement succeeds, the read still fails with
    > the import error.


    The unpickle code uses something morally equivalent to __import__ to
    convert the module name in the pickled data stream into a module
    object. The symbols defined in the module you called pickle.load from
    don't matter at all, just like any other function defined in a separate
    module.

    Pickle isn't meant for transmitting data between two unrelated
    programs. It's designed to be a low-development-effort way to preserve
    and restore objects for a single program or a group of related programs
    that share libraries. If you want to share data between unrelated
    programs, use something a little more interoperable - json, xml, or
    some other program-agnostic representation.

    -o

    (And if you were thinking of using pickle over the internet, forget it:
    the pickle encoding is a small stack-based programming language in its
    own right, and can be used to invoke semi-arbitrary module-scoped
    funtions.)
    Owen Jacobson, May 4, 2011
    #3
  4. Catherine Moroney wrote:

    > I am having some problems reading the
    > object back out, as I get complaints about "unable to import module X".
    >
    > The only way I have found around it is to run the read-file code out of
    > the same directory that contains the X.py file
    >
    > Even when I put statements into the code such as "from Y.X import X" ...
    > the import statement works, but I am still unable to read the object


    Is the program that reads the pickle file the same one that
    was used to write it?

    If not, what might be happening is that the writing program
    has module X at the top level instead of inside package Y.
    Then it will get pickled simply under the name "X" instead
    of "Y.X", and the reading program will expect to find it at
    the top level as well.

    It's important that the reading and writing programs agree
    about the location of the pickled classes in the package
    namespace, unless you take steps to customise the pickling
    process.

    --
    Greg
    Gregory Ewing, May 5, 2011
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ryan Grow
    Replies:
    1
    Views:
    333
    Tim Keating
    Nov 16, 2004
  2. Bram Stolk
    Replies:
    0
    Views:
    249
    Bram Stolk
    Dec 12, 2005
  3. krishnakant Mane
    Replies:
    2
    Views:
    221
    Bruno Desthuilliers
    May 4, 2007
  4. krishnakant Mane
    Replies:
    2
    Views:
    264
    Daniele Varrazzo
    May 7, 2007
  5. Jean-Paul Calderone

    Re: Pickled objects over the network

    Jean-Paul Calderone, Jul 18, 2007, in forum: Python
    Replies:
    3
    Views:
    715
    Irmen de Jong
    Jul 19, 2007
Loading...

Share This Page