numarray + cPickle problem

[Bill Mill]

Hello all,

I've pickled a numarray array object using cPickle like so:

pickle = cPickle.Pickler(fout, -1)
pickle.dump((myarray, list1, list2))

and this seems to work fine, until I try to load the array back into
memory:

up = cPickle.Unpickler(fin)
myarray, list1, list2 = up.load()

which causes an odd traceback:

Traceback (most recent call last):
File "do_lsi.py", line 7, in ?
data, artists, users = up.load()
ImportError: No module named generic

To which I say, huh? There is indeed a numarray.generic module - but
numarray is already included into this file. I tried including
numarray.generic, or "from numarray import generic" or "from
numarray.generic import *", but none of those did anything.

Anybody able to guess what is happeneing here? Anybody that can explain
the unpickling process in greater detail for me so that I may
understand it better?

Peace
Bill Mill
bill.mill at gmail.com

 

[Bill Mill]

A couple more details:

I'm using Python 2.3.4 and numarray 1.1. I've now found the
numarray.array_persist module, which will let me pickle numarrays an
alternative way, but I'd still like to know what's going on here. I can
definitely pickle numarray.array objects in trivial cases.

Why doesn't Python raise an Unpickleable error? Why is it trying to
import generic?

 

[Bill Mill]

Ok, sorry for sending another message to the list so soon, but I think
I've figured out the problem. On Windows, if you want to use one of the
binary pickle settings, you *need* to open your output file in binary
mode. Perhaps this should be mentioned in the cPickle documentation?
Peace
Bill Mill
bill.mill at gmail.com

 

[Andrea Griffini]

Perhaps this should be mentioned in the cPickle documentation?

It *IS* documented...

Pickler(...)
Pickler(file, proto=0) -- Create a pickler.

This takes a file-like object for writing a pickle
data stream.
The optional proto argument tells the pickler to
use the given protocol; supported protocols are 0,
1, 2. The default protocol is 0, to be backwards
compatible. (Protocol 0 is the only protocol that
can be written to a file opened in text mode and
--> read back successfully. When using a protocol
--> higher than 0, make sure the file is opened in
--> binary mode, both when pickling and unpickling.)

...

Andrea

 

[Bill Mill]

Andrea,

It *IS* documented...

I saw that when I figured out what was going on. I should have been
more specific. What I meant was that in the Python module documentation
(of cPickle or pickle) on the web should include that little tidbit
which is really rather important. I don't believe that you need to use
binary mode on Unix (though I could be wrong), and that is where I am
used to working, so this error surprised me.

On the website, it doesn't say that anywhere, and it should, IMHO.
Peace
Bill Mill
bill.mill at gmail.com

 

[Andreas Kostyrka]

A couple more details:

I'm using Python 2.3.4 and numarray 1.1. I've now found the
numarray.array_persist module, which will let me pickle numarrays an
alternative way, but I'd still like to know what's going on here. I can
definitely pickle numarray.array objects in trivial cases.

Why doesn't Python raise an Unpickleable error? Why is it trying to
import generic?

Just a guess: the generic submodule is a C language module that doesn't
know that it's meant to be a submodule. -> So it believes it's just
"generic.numarray" and not correctly "numpy.generic.numarray".

Andreas