app runs fine with interpreter, but not under py2exe

D

Doug Morse

Hi,

I have an application that runs just fine using the standard Python distro
interpreter (v2.5.1 on WinXP) but throws the following exception when run as
an executable built with py2exe. My questions are: (a) does anyone have any
thoughts on why this exception is occurring and what to do about it, and (b)
more generally, why would a problem like this occur under py2exe but not with
the standard distro?

Thanks in adavance for any and all help.

Cheers,
Doug


Traceback (most recent call last):
File "VisionTrainer.py", line 49, in <module>
File "SessionController.pyc", line 53, in <module>
File "VisionEgg\__init__.pyc", line 42, in <module>
File "VisionEgg\ParameterTypes.pyc", line 28, in <module>
File "Numeric.pyc", line 93, in <module>
File "Precision.pyc", line 26, in <module>
File "Precision.pyc", line 23, in _fill_table
File "Precision.pyc", line 18, in _get_precisions
TypeError: data type not understood
 
G

GHUM

Doug,
File "VisionTrainer.py", line 49, in <module>
File "SessionController.pyc", line 53, in <module>
File "VisionEgg\__init__.pyc", line 42, in <module>
File "VisionEgg\ParameterTypes.pyc", line 28, in <module>
File "Numeric.pyc", line 93, in <module>
File "Precision.pyc", line 26, in <module>
File "Precision.pyc", line 23, in _fill_table
File "Precision.pyc", line 18, in _get_precisions
TypeError: data type not understood

to my knowledge, "data type not understood" is a message not from core
Python, but rather thrown from "your" code. What is happening around
Precision.py, line 18?

What does trigger that exception?

My guess is, you are dealing with some custom imported modules which
define "data type"s.
(PIL does something similiar for supported images)... that is, some
module is trying to import all definitions from a specific directory.

That "dynamic importing" fails within py2exe --- all the importing has
to be definit at py2exing-time. Please ready http://www.py2exe.org/index.cgi/PIL_and_py2exe
and the other receipe on py2exe.org and try to adapt that knowledge to
your application.

Harald

Harald
 
D

Doug Morse

Hi Harald and C.L.P.,

Precision.py is part of the Numeric package. AFAIKT, the problem is during
the module initialization. The first lines of Precision.py are:

from multiarray import zeros
import string

typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
'Float':'fd', 'Complex':'FD'}

def _get_precisions(typecodes):
lst = []
for t in typecodes:
lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18
return lst

def _fill_table(typecodes, table={}):
for key, value in typecodes.items():
table[key] = _get_precisions(value)
return table

_code_table = _fill_table(typecodes)


I can't find any reason why line 18 is throwing a "data type not understood"
error. It doesn't seem to have anything to do with dynamic importing, but I
can't be sure. I would note that "zeros" is a built-in function found in the
"python dll" multiarray.pyd (in the Numeric module directory).

Thanks again,
Doug
 
G

GHUM

Doug,
Precision.py is part of the Numeric package. AFAIKT, the problem is during
the module initialization. The first lines of Precision.py are:

from multiarray import zeros
import string
[.....]
and your program is crashing on the

lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18

line... Please, try the following:

import multiarray
from multiarray import zeros
import string

(adding the line "import multiarray" before zeros are imported from
it)

I used this workaround with various packages I py2exed - maybe it also
works for you?

please keep us updated,

Harald
 
D

Doug Morse

Harald,

Great suggestion, thanks! Unfortunately, "no help there". Adding "import
multiarray" to Precision.py where you suggested (i.e., before the "from
multiarray import zeros" line) did not fix the problem. I'm getting the same
exception and traceback as before except, of course, on line 19 now instead of
line 18.

Doug


Doug,
Precision.py is part of the Numeric package. AFAIKT, the problem is during
the module initialization. The first lines of Precision.py are:

from multiarray import zeros
import string
[.....]
and your program is crashing on the

lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18

line... Please, try the following:

import multiarray
from multiarray import zeros
import string

(adding the line "import multiarray" before zeros are imported from
it)

I used this workaround with various packages I py2exed - maybe it also
works for you?

please keep us updated,

Harald
 
P

Peter Otten

Doug said:
from multiarray import zeros
import string

typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
'Float':'fd', 'Complex':'FD'}

def _get_precisions(typecodes):
    lst = []
    for t in typecodes:
        lst.append( (zeros( (1,), t ).itemsize()*8, t) )   <-- Line 18
    return lst

def _fill_table(typecodes, table={}):
    for key, value in typecodes.items():
        table[key] = _get_precisions(value)
    return table

_code_table = _fill_table(typecodes)

I can't find any reason why line 18 is throwing a "data type not
understood" error.  It doesn't seem to have anything to do with dynamic
importing, but I can't be sure.  I would note that "zeros" is a built-in
function found in the "python dll" multiarray.pyd (in the Numeric module
directory).

I don't know why, but your module seems to use numpy's multiarray instead of
Numeric's:
from multiarray import zeros
zeros((1,), "1") array([0], '1')
from numpy.core.multiarray import zeros
zeros((1,), "1")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: data type not understood

Peter
 
D

Doug Morse

Peter,

Genius! You nailed it -- thanks!

py2exe is apparently getting confused by the fact that packages "Numeric" and
"numpy" both have files multiarray.pyd and umath.pyd. It copies just one of
each -- from $PYTHONHOME/Lib/site-packages/numpy/core -- and puts both of them
into the top-level of the created "dist" directory. Also, there are no such
files as multiarray.pyc and umath.pyc in my python installation, but py2exe
seems to create these (in the dist and dist/numpy/core directories) -- they
are small (e.g., 526 bytes) and I'm guessing that they are stubs that simply
point python to the matching .pyd that py2exe relocated.

So, by using the --skip-archive option to py2exe (i.e., python setup.py py2exe
--skip-archive) to create the dist and build directories, and then within the
dist directory hierarchy removing all the occurrances of multiarray.pyc and
umath.pyc, and then moving dist/multiarray.pyd and dist/umath.pyd to the
dist/numpy/core directory, and finally by copying multiarray.pyd and umath.pyd
from $PYTHONHOME/Lib/site-packages/Numeric to the dist directory, I am able to
get my application to run correctly and as a standalone executable. HOORAH!!!

Just for completeness and perhaps a bit of additional clarity, here's a
limited directory listing of what the steps in the previous paragraph produce:

morse> find dist | egrep "(multia|umath)" | xargs ls -l
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
morse>

(This is still WinXP; I'm just using the Cygwin bash and fileutils here.
Also, note that there are NO multiarray.pyc or umath.pyc files.)

So, my next step will be to try to not use the --skip-archive option and then
make these same modifications regarding multiarray.pyd and umath.pyd to the
py2exe-generated library.zip file and see if I can get things running that way
as well (and in so doing reduce my "dist" directory by about 10mg). I may
also try creating a dist/Numeric subdirectory and moving dist/multiarray.pyd
and dist/umath.pyd to this dist/Numeric subdirectory -- for the goal of more
accurately mirroring the actual file/directory structure found in
$PYTHONHOME/Lib/site-packages.

Again, thanks to everyone for their assistance, esp. Harald and Peter.

Is this something I should pursue getting the py2exe folks to fix / work with
them on fixing? In investigating this issue, I noted that a lot of other
py2exe problems seem to revolve around confusions when duplicate files exist
in different modules. I wouldn't thinking that getting py2exe to pay better
attention to the containing modules when building it's dependency tree would
be that difficult (or is it)?

Cheers,
Doug


Doug said:
from multiarray import zeros
import string

typecodes = {'Character':'c', 'Integer':'1sil', 'UnsignedInteger':'bwu',
'Float':'fd', 'Complex':'FD'}

def _get_precisions(typecodes):
    lst = []
    for t in typecodes:
        lst.append( (zeros( (1,), t ).itemsize()*8, t) )   <-- Line 18
    return lst

def _fill_table(typecodes, table={}):
    for key, value in typecodes.items():
        table[key] = _get_precisions(value)
    return table

_code_table = _fill_table(typecodes)

I can't find any reason why line 18 is throwing a "data type not
understood" error.  It doesn't seem to have anything to do with dynamic
importing, but I can't be sure.  I would note that "zeros" is a built-in
function found in the "python dll" multiarray.pyd (in the Numeric module
directory).

I don't know why, but your module seems to use numpy's multiarray instead of
Numeric's:
from multiarray import zeros
zeros((1,), "1") array([0], '1')
from numpy.core.multiarray import zeros
zeros((1,), "1")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: data type not understood

Peter
 
D

Doug Morse

Hi,

Well, my attempt to not use the --skip-archive option didn't get very far,
as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
I'm guessing that they can't be in library.zip for a reason (i.e., they are
DLL files, essentially, and thus must be readily available to be loaded
into memory).

Is there a work-around for this, then? That is, is there a way to either
(a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
umath.pyd files or (b) perhaps rename one set of the .pyd files -- say the
numpy/core versions -- to say multiarray2.pyd and umath2.pyd, and then
manual create the "stub"-like .pyc files that py2exe creates to point to
these alternate .pyd files and then place these stubs in
library.zip/numpy/core? Or am I just hoping for too much here and am going
to be stuck with using the --skip-archive option?

Thanks,
Doug
 
G

GHUM

Doug,
as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
I'm guessing that they can't be in library.zip for a reason (i.e., they are
DLL files, essentially, and thus must be readily available to be loaded
into memory).

.dll and .pyd files CAN be within library.zip and even within the
single-file-exe.

There is some Hellermagic within _memimporter.pyd that loads the .dll
and .pyd from a zipfile. When you scan the build_exe.py, you will
explicitely find:

print "*** finding dlls needed ***"
dlls = self.find_dlls(extensions)
self.plat_finalize(mf.modules, py_files, extensions, dlls)
dlls = [item for item in dlls
if os.path.basename(item).lower() not in
self.dll_excludes]
# should we filter self.other_depends in the same way?
Is there a work-around for this, then?
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/
multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/
umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd

let's try: your mission is, should you chose to accept it, to make a
py2exed application to load two different multiarray.pyd, who are on
the visible level only different by their place in the file system.

So, please make a minimal script:

import multiarray
import numpy.core.multiarray as ncmultiarray

and, using Peters testassert zeros((1,), "1") == array([0], '1')

and check, if those multiarrays are the correct ones. If yes, you can
try to build upon this by making all imports explicit as described
above. (Yes, I know, that's not necessary in plain python ... but it
saved me in some weird "import from .zip" and "import from database"
situations.)
(a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
umath.pyd files

somewhere deep inside I have a BAAAD feeling about to files having the
same name but very very different content - but that's not your call
in this situation.
Are you able to build that libraries yourself? Then you could move
down and build the multiarray.pyd as ncmultiarray.pyd and adjust all
imports accordingly. (lots of work)
manual create the "stub"-like .pyc files that py2exe creates to point to
these alternate .pyd files and then place these stubs in
library.zip/numpy/core?  

As much as I know there is only one central "loader stub", which
ensures that .dlls get imported from memory, see above. py2exe does
not create something like p2elma.py containing "import multiarray.pyd"

best luck, and please take your time to scan the py2exe.org wiki,
there are many similiar challanges with various packages, maybe you
get another workaround-idea (and, if we find one, PLEASE put it down
there for others to learn)

best wishes,

Harald
 
D

Doug Morse

Hi Harald,

Bond here, James Bond. I accepted your mission. :)

Unfortunately, the mission failed. Creating a "testapp.py" as you described
(i.e., with the suggested import statements) runs just fine with the
interpreter. again, however, py2exe does the same thing as the original
problem -- that is, copies numpy.core's versions of multiarray.pyd and
umath.pyd into dist/, creates what look like stub multiarray.pyc and umath.pyc
files in library.zip/ and library.zip/numpy/core/. when this py2exe version
is run, it throws the same (original) TypeError: data type not understood
error.

Also, with either my original program or testapp.py, I cannot seem to modify
library.zip in such a way as to get the program to run. Basically, I unzipped
library.zip, made the same modifications to it's contents as I did when using
the --skip-archive option (i.e., remove the stub files and put the right .pyd
files in the right locations) and then recreated the library.zip file.
Afterwards, running either origapp.exe or testapp.exe, the program bombs out
with ImportError: no module named multiarray.

In addition, I tried using py2exe to make a simple "Hello, world" program. It
runs just fine. Then, I "zip -m library.zip *.pyd" in the dist/ directory to
move the .pyd files into library.zip. Then, running hello.exe again works
just fine. However, if I add "import bz2; print bz2.__author__" after "print
'Hello, world!'" (the only line in hello.py), hello.exe bombs out with
"ImportError: DLL load failed: The specified module could not be found" when I
run it after having moved *.pyd into library.zip/. (bz2 was one of only two
..pyd files that py2exe put into dist/.)

So, I can seem to figure out how to get .pyd files into library.zip, even when
there are no name conflicts. What am I doing wrong?

Finally, regarding the stub files, if I use vi to edit the py2exe-generated,
536-byte bz2.pyc file (there is no such file in my python install directory
hierarchy), I see the word "import" very close to the word "bz2.pyd". This
suggests to me that py2exe might indeed be creating simple import statements
in a short .py file for each .pyd file and then compiling it into a .pyc stub
file -- or at least doing something similar. Of course I can't be sure of
this, but it seems likely to me.

Thanks again to you and everyone. I'll definitely visit the py2exe wiki and
see what I can come up with (and or course will report back with success /
failures). If you or anyone has any further thoughts based on this post, I'm
all ears and would be most grateful.

Regards,
Doug


Doug,
as I quickly noticed that "library.zip" does NOT contain ANY .pyd files.
I'm guessing that they can't be in library.zip for a reason (i.e., they are
DLL files, essentially, and thus must be readily available to be loaded
into memory).

.dll and .pyd files CAN be within library.zip and even within the
single-file-exe.

There is some Hellermagic within _memimporter.pyd that loads the .dll
and .pyd from a zipfile. When you scan the build_exe.py, you will
explicitely find:

print "*** finding dlls needed ***"
dlls = self.find_dlls(extensions)
self.plat_finalize(mf.modules, py_files, extensions, dlls)
dlls = [item for item in dlls
if os.path.basename(item).lower() not in
self.dll_excludes]
# should we filter self.other_depends in the same way?
Is there a work-around for this, then?
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/
multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/
umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd

let's try: your mission is, should you chose to accept it, to make a
py2exed application to load two different multiarray.pyd, who are on
the visible level only different by their place in the file system.

So, please make a minimal script:

import multiarray
import numpy.core.multiarray as ncmultiarray

and, using Peters testassert zeros((1,), "1") == array([0], '1')

and check, if those multiarrays are the correct ones. If yes, you can
try to build upon this by making all imports explicit as described
above. (Yes, I know, that's not necessary in plain python ... but it
saved me in some weird "import from .zip" and "import from database"
situations.)
(a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
umath.pyd files

somewhere deep inside I have a BAAAD feeling about to files having the
same name but very very different content - but that's not your call
in this situation.
Are you able to build that libraries yourself? Then you could move
down and build the multiarray.pyd as ncmultiarray.pyd and adjust all
imports accordingly. (lots of work)
manual create the "stub"-like .pyc files that py2exe creates to point to
these alternate .pyd files and then place these stubs in
library.zip/numpy/core?  

As much as I know there is only one central "loader stub", which
ensures that .dlls get imported from memory, see above. py2exe does
not create something like p2elma.py containing "import multiarray.pyd"

best luck, and please take your time to scan the py2exe.org wiki,
there are many similiar challanges with various packages, maybe you
get another workaround-idea (and, if we find one, PLEASE put it down
there for others to learn)

best wishes,

Harald
 
T

Thomas Heller

Doug said:
Peter,

Genius! You nailed it -- thanks!

py2exe is apparently getting confused by the fact that packages "Numeric" and
"numpy" both have files multiarray.pyd and umath.pyd. It copies just one of
each -- from $PYTHONHOME/Lib/site-packages/numpy/core -- and puts both of them
into the top-level of the created "dist" directory. Also, there are no such
files as multiarray.pyc and umath.pyc in my python installation, but py2exe
seems to create these (in the dist and dist/numpy/core directories) -- they
are small (e.g., 526 bytes) and I'm guessing that they are stubs that simply
point python to the matching .pyd that py2exe relocated. [...]

Just for completeness and perhaps a bit of additional clarity, here's a
limited directory listing of what the steps in the previous paragraph produce:

morse> find dist | egrep "(multia|umath)" | xargs ls -l
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
[...]
Is this something I should pursue getting the py2exe folks to fix / work with
them on fixing? In investigating this issue, I noted that a lot of other
py2exe problems seem to revolve around confusions when duplicate files exist
in different modules. I wouldn't thinking that getting py2exe to pay better
attention to the containing modules when building it's dependency tree would
be that difficult (or is it)?

I have the impression that this issue may already be fixed in py2exe CVS but
noone bothered to do a release. Here is the corresponding changelog entry:

* build_exe.py: Patch from Grant Edwards, slightly adjusted:
py2exe now renames the pyd files that it copies into the dist
directory so that they include the package name. This prevents
name conflicts.

I have created installers and uploaded them (for testing!) to the starship:

http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.3.exe
http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.4.exe
http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.5.exe

Please try them out.

Thanks,
Thomas
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top