importing class objects from a pickled file

  • Thread starter Catherine Moroney
  • Start date
C

Catherine Moroney

Hello,

I have an object of class X that I am writing to a pickled file. The
pickling part goes fine, but I am having some problems reading the
object back out, as I get complaints about "unable to import module X".

The only way I have found around it is to run the read-file code out of
the same directory that contains the X.py file, but this is obviously
not a portable way of doing things.

Even when I put statements into the code such as "from Y.X import X"
where Y is the name of the python package that contains the X,py file,
the import statement works, but I am still unable to read the object
from the pickled file, running into the same "unable to import module X"
error.

Am I explaining myself properly? Why doesn't the code that loads the
object from the pickled file work unless I am sitting in the same
directory? The code that writes the pickled file has the statement
"from Y.X import X" statement" at the top, as does the reading code, but
even though that import statement succeeds, the read still fails with
the import error.

Thanks for any help,

Catherine
 
J

James Mills

Am I explaining myself properly?  Why doesn't the code that loads the
object from the pickled file work unless I am sitting in the same directory?
  The code that writes the pickled file has the statement
"from Y.X import X" statement" at the top, as does the reading code, but
even though that import statement succeeds, the read still fails with the
import error.

pickled objects won't work this way unless you "customize" the
way in which your class gets serialized.

See: http://docs.python.org/library/pickle.html#the-pickle-protocol

Any time you want to unpickle a user class, that class must be available.

I suggest serializing to a more common format (say JSON) and re-create
your class with the data.

cheers
James
 
O

Owen Jacobson

Hello,

I have an object of class X that I am writing to a pickled file. The
pickling part goes fine, but I am having some problems reading the
object back out, as I get complaints about "unable to import module X".

The only way I have found around it is to run the read-file code out of
the same directory that contains the X.py file, but this is obviously
not a portable way of doing things.

Even when I put statements into the code such as "from Y.X import X"
where Y is the name of the python package that contains the X,py file,
the import statement works, but I am still unable to read the object
from the pickled file, running into the same "unable to import module X"
error.

Am I explaining myself properly? Why doesn't the code that loads the
object from the pickled file work unless I am sitting in the same directory?

That's not the actual requirement.

The pickled representation of an instance of a user-defined class contains:

* the name of the class
* the name of the class's module
* the object's attributes, pickled recursively

You can have a look at the representation using protocol zero, which is
printable and not terribly hard to read. Here's an example, lightly
annotated:
... def __init__(self, x):
... self.x = x
...ccopy_reg
_reconstructor
p0
module name (c__main__
class name Example
p1
c__builtin__
object
p2
Ntp3
Rp4
(dp5
field name S'x'
p6
field value I1
sb.
Obviously, since the pickled representation only contains the *name and
location* of the class and not the class itself, you need to have the
definition of the class handy when unpickling an object. This means
that the module that defines the class must be on the module search
path - either because it's in the current directory (or a package in
the current directory, or..), or because it's in PYTHONPATH, or because
it's somewhere else on sys.path.
The code that writes the pickled file has the statement
"from Y.X import X" statement" at the top, as does the reading code, but
even though that import statement succeeds, the read still fails with
the import error.

The unpickle code uses something morally equivalent to __import__ to
convert the module name in the pickled data stream into a module
object. The symbols defined in the module you called pickle.load from
don't matter at all, just like any other function defined in a separate
module.

Pickle isn't meant for transmitting data between two unrelated
programs. It's designed to be a low-development-effort way to preserve
and restore objects for a single program or a group of related programs
that share libraries. If you want to share data between unrelated
programs, use something a little more interoperable - json, xml, or
some other program-agnostic representation.

-o

(And if you were thinking of using pickle over the internet, forget it:
the pickle encoding is a small stack-based programming language in its
own right, and can be used to invoke semi-arbitrary module-scoped
funtions.)
 
G

Gregory Ewing

Catherine said:
I am having some problems reading the
object back out, as I get complaints about "unable to import module X".

The only way I have found around it is to run the read-file code out of
the same directory that contains the X.py file

Even when I put statements into the code such as "from Y.X import X" ...
the import statement works, but I am still unable to read the object

Is the program that reads the pickle file the same one that
was used to write it?

If not, what might be happening is that the writing program
has module X at the top level instead of inside package Y.
Then it will get pickled simply under the name "X" instead
of "Y.X", and the reading program will expect to find it at
the top level as well.

It's important that the reading and writing programs agree
about the location of the pickled classes in the package
namespace, unless you take steps to customise the pickling
process.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top