"Bad file descriptor" after py2exe or Installer

[Luc Saffre]

Hello,

I had a strange problem when freezing (using either py2exe or McMillan
installer) a script that imports reportlab (which imports PIL (which
imports FixTk))).

- Python 2.3.3c (also with Python 2.3)
- PIL 1.1.4
- Installer or py2exe : latest versions.

Here is how I can reproduce it on a Win2000 and on a Win XP machine. I
also found a hack that works for me, but have no idea why all this happens.

The following script `test.py` runs without problem when interpreted:

print "before"
from PIL import Image
print "after"

Output as expected:

c:\test>python test.py
before
after
c:\test>

But if I freeze it and invoke the `test.exe`, I get a strange
behaviour::

c:\test\dist> test
before
c:\test\dist>

The second print just does not happen. And if I redirect stderr to a
file, I see a bit more::

c:\test\dist> test 2> tmp.txt
before
c:\test\dist> type tmp.txt
Traceback (most recent call last):
File "test.py", line 3, in ?
print "after"
IOError: [Errno 9] Bad file descriptor
c:\test\dist>

Workaround:
If I remove the following lines (41-46) of PIL/Image.py::

try:
# give Tk a chance to set up the environment, in case we're
# using an _imaging module linked against libtcl/libtk
import FixTk
except ImportError:
pass

then the problem disappears, and both script and executable work for
me...

If somebody sees an explanation, please let me know!

Luc Saffre

 

[Luc Saffre]

> It seems I cannot reproduce this with py2exe 0.5.
> Can you?

Yes: I use py2exe 0.5.0a5.

Here is in more detailed, how I reproduce the problem:

I used the following installers:
Python-2.3.3c1.exe
PIL-1.1.4.win32-py2.3.exe
py2exe-0.5.0a5.win32-py2.3.exe
I installed these three to a clean installation (except for my windows
registry where I don't know how to verify cleaness)

I place the test.py as described into a separate directory, together
with a file setup.py:

from distutils.core import setup
import py2exe
setup( console=["test.py"] )

and then I type "setup py2exe" at the command line.
Here is the output of that command:

running py2exe
*** searching for required modules ***
*** parsing results ***
creating python loader for extension '_sre'
creating python loader for extension 'PIL._imaging'
creating python loader for extension '_tkinter'
creating python loader for extension 'PIL._imagingtk'
*** finding dlls needed ***
*** create binaries ***
*** byte compile python files ***
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\PaletteFile.py to PIL\PaletteFile.pyc
skipping byte-compilation of C:\PYTHON23\lib\lib-tk\FixTk.py to FixTk.pyc
skipping byte-compilation of C:\PYTHON23\lib\copy_reg.py to copy_reg.pyc
skipping byte-compilation of C:\PYTHON23\lib\sre_compile.py to
sre_compile.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImageChops.py to PIL\ImageChops.pyc
skipping byte-compilation of C:\PYTHON23\lib\StringIO.py to StringIO.pyc
byte-compiling
T:\data\luc\release\lino\tests\py2exe\build\bdist.win32\winexe\temp\_sre.py
to _sre.pyc
skipping byte-compilation of C:\PYTHON23\lib\macpath.py to macpath.pyc
skipping byte-compilation of C:\PYTHON23\lib\popen2.py to popen2.pyc
skipping byte-compilation of C:\PYTHON23\lib\atexit.py to atexit.pyc
skipping byte-compilation of C:\PYTHON23\lib\dummy_thread.py to
dummy_thread.pyc
byte-compiling
T:\data\luc\release\lino\tests\py2exe\build\bdist.win32\winexe\temp\PIL._imaging.py
to PIL\_imaging.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImageFilter.py to PIL\ImageFilter.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\GimpGradientFile.py to
PIL\GimpGradientFile.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\__init__.py to PIL\__init__.pyc
skipping byte-compilation of C:\PYTHON23\lib\posixpath.py to posixpath.pyc
byte-compiling
T:\data\luc\release\lino\tests\py2exe\build\bdist.win32\winexe\temp\_tkinter.py
to _tkinter.pyc
skipping byte-compilation of C:\PYTHON23\lib\linecache.py to linecache.pyc
skipping byte-compilation of C:\PYTHON23\lib\tempfile.py to tempfile.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\GimpPaletteFile.py to
PIL\GimpPaletteFile.pyc
skipping byte-compilation of C:\PYTHON23\lib\sre_constants.py to
sre_constants.pyc
skipping byte-compilation of C:\PYTHON23\lib\re.py to re.pyc
skipping byte-compilation of C:\PYTHON23\lib\ntpath.py to ntpath.pyc
skipping byte-compilation of C:\PYTHON23\lib\stat.py to stat.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImagePalette.py to PIL\ImagePalette.pyc
skipping byte-compilation of C:\PYTHON23\lib\lib-tk\Tkinter.py to
Tkinter.pyc
skipping byte-compilation of C:\PYTHON23\lib\string.py to string.pyc
skipping byte-compilation of C:\PYTHON23\lib\warnings.py to warnings.pyc
skipping byte-compilation of C:\PYTHON23\lib\UserDict.py to UserDict.pyc
skipping byte-compilation of C:\PYTHON23\lib\site-packages\PIL\Image.py
to PIL\Image.pyc
skipping byte-compilation of C:\PYTHON23\lib\repr.py to repr.pyc
byte-compiling
T:\data\luc\release\lino\tests\py2exe\build\bdist.win32\winexe\temp\PIL._imagingtk.py
to PIL\_imagingtk.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImageTk.py to PIL\ImageTk.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImageTransform.py to
PIL\ImageTransform.pyc
skipping byte-compilation of C:\PYTHON23\lib\copy.py to copy.pyc
skipping byte-compilation of
C:\PYTHON23\lib\site-packages\PIL\ImageColor.py to PIL\ImageColor.pyc
skipping byte-compilation of C:\PYTHON23\lib\types.py to types.pyc
skipping byte-compilation of C:\PYTHON23\lib\sre.py to sre.pyc
skipping byte-compilation of C:\PYTHON23\lib\traceback.py to traceback.pyc
skipping byte-compilation of C:\PYTHON23\lib\lib-tk\Tkconstants.py to
Tkconstants.pyc
skipping byte-compilation of C:\PYTHON23\lib\random.py to random.pyc
skipping byte-compilation of C:\PYTHON23\lib\os2emxpath.py to os2emxpath.pyc
skipping byte-compilation of C:\PYTHON23\lib\colorsys.py to colorsys.pyc
skipping byte-compilation of C:\PYTHON23\lib\sre_parse.py to sre_parse.pyc
skipping byte-compilation of C:\PYTHON23\lib\os.py to os.pyc
*** copy extensions ***
copying C:\PYTHON23\DLLs\_sre.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\PIL\_imaging.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\DLLs\_tkinter.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\PIL\_imagingtk.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
*** copy dlls ***
copying u:\CYGWIN\BIN\cygwin1.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying u:\CYGWIN\BIN\tk84.dll -> T:\data\luc\release\lino\tests\py2exe\dist
copying C:\WINNT\System32\python23.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\DLLs\tcl84.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying u:\CYGWIN\BIN\tcl84.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\py2exe\run.exe ->
T:\data\luc\release\lino\tests\py2exe\dist\test.exe
copying C:\PYTHON23\lib\site-packages\py2exe\run_w.exe ->
T:\data\luc\release\lino\tests\py2exe\dist\tkhello.exe

Afterwards the dist\test.exe shows the described behavious.

I hope that you (or somebody else) can now reproduce the problem.

TIA,
Luc

 

[Thomas Heller]

It seems I cannot reproduce this with py2exe 0.5.
Can you?

Yes: I use py2exe 0.5.0a5.

Here is in more detailed, how I reproduce the problem:

I used the following installers:
Python-2.3.3c1.exe
PIL-1.1.4.win32-py2.3.exe
py2exe-0.5.0a5.win32-py2.3.exe
I installed these three to a clean installation (except for my windows
registry where I don't know how to verify cleaness)

I place the test.py as described into a separate directory, together
with a file setup.py:

from distutils.core import setup
import py2exe
setup( console=["test.py"] )

and then I type "setup py2exe" at the command line.
Here is the output of that command:
[...]
*** copy extensions ***
copying C:\PYTHON23\DLLs\_sre.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\PIL\_imaging.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\DLLs\_tkinter.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\PIL\_imagingtk.pyd ->
T:\data\luc\release\lino\tests\py2exe\dist
*** copy dlls ***

I guess the problem is here:

copying u:\CYGWIN\BIN\cygwin1.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying u:\CYGWIN\BIN\tk84.dll -> T:\data\luc\release\lino\tests\py2exe\dist
copying C:\WINNT\System32\python23.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\DLLs\tcl84.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying u:\CYGWIN\BIN\tcl84.dll ->
T:\data\luc\release\lino\tests\py2exe\dist
copying C:\PYTHON23\lib\site-packages\py2exe\run.exe ->
T:\data\luc\release\lino\tests\py2exe\dist\test.exe
copying C:\PYTHON23\lib\site-packages\py2exe\run_w.exe ->
T:\data\luc\release\lino\tests\py2exe\dist\tkhello.exe

py2exe seems to pick up cygwin binaries. The reason could be that
either you have cygwin extensions somewhere in your pythonpath, or parts
of the PATH env variable points into the cygwin directories, or
something like that. You should look at the output above carefully, and
make sure that it is what one would expect.

HTH,

Thomas

 

[Luc Saffre]

py2exe seems to pick up cygwin binaries. The reason could be that
either you have cygwin extensions somewhere in your pythonpath, or parts
of the PATH env variable points into the cygwin directories, or
something like that. You should look at the output above carefully, and
make sure that it is what one would expect.

That's it! Thank you, Thomas, your hint helped.

I had the Cygwin tcltk package installed. After removing this package
(using cygwin's setup.exe), I can use again the original PIL/Image.py
and everything works fine.

My PATH contains the cygwin bin directories:
PATH=u:\CYGWIN\BIN;
c:\python23\Scripts;c:\python23;
....
u:\CYGWIN\USR\BIN;u:\CYGWIN\USR\LOCAL\BIN;...

Another workaround is to remove the cygwin directories from PATH:
I reinstalled the cygwin tcltk and the problem reappeared,
moving the first cygwin directory to come after the python directory did
not help. Then I removed those directories completely from PATH (during
the build) and the problem disappeared again.

I would consider this as a bug (in both py2exe and Installer) since they
decide to pick some cygwin tcl/tk dll's who happen to hang around while
the Python runtime won't...

Luc

 

[Thomas Heller]

That's it! Thank you, Thomas, your hint helped.

I had the Cygwin tcltk package installed. After removing this package
(using cygwin's setup.exe), I can use again the original PIL/Image.py
and everything works fine.

My PATH contains the cygwin bin directories:
PATH=u:\CYGWIN\BIN;
c:\python23\Scripts;c:\python23;
...
u:\CYGWIN\USR\BIN;u:\CYGWIN\USR\LOCAL\BIN;...

Another workaround is to remove the cygwin directories from PATH:
I reinstalled the cygwin tcltk and the problem reappeared,
moving the first cygwin directory to come after the python directory
did not help. Then I removed those directories completely from PATH
(during the build) and the problem disappeared again.

I would consider this as a bug (in both py2exe and Installer) since
they decide to pick some cygwin tcl/tk dll's who happen to hang around
while the Python runtime won't...

Yep, that's a bug.

Although I wonder: does it really make sense to include cygwin's path
into PATH? Isn't cygwin only usefull from within the cygwin prompt?

I tried cygwin some time ago, and was initially broken because I had
installed (native) mingw32 somewhere, so it seemed to me to better keep
cmd.exe and cygwin's paths separate from each other.

Thomas

 

[Thomas Heller]

[posted and mailed]

That's it! Thank you, Thomas, your hint helped.

I had the Cygwin tcltk package installed. After removing this package
(using cygwin's setup.exe), I can use again the original PIL/Image.py
and everything works fine.

My PATH contains the cygwin bin directories:
PATH=u:\CYGWIN\BIN;
c:\python23\Scripts;c:\python23;
...
u:\CYGWIN\USR\BIN;u:\CYGWIN\USR\LOCAL\BIN;...

Another workaround is to remove the cygwin directories from PATH:
I reinstalled the cygwin tcltk and the problem reappeared,
moving the first cygwin directory to come after the python directory
did not help. Then I removed those directories completely from PATH
(during the build) and the problem disappeared again.

I would consider this as a bug (in both py2exe and Installer) since
they decide to pick some cygwin tcl/tk dll's who happen to hang around
while the Python runtime won't...

This is part of the code (in build_exe.py).

def find_dependend_dlls(self, use_runw, dlls, pypath, dll_excludes):
import py2exe_util
sysdir = py2exe_util.get_sysdir()
windir = py2exe_util.get_windir()
# This is the tail of the path windows uses when looking for dlls
# XXX On Windows NT, the SYSTEM directory is also searched
exedir = os.path.dirname(sys.executable)
syspath = os.environ['PATH']
loadpath = ';'.join([exedir, sysdir, windir, syspath])

# Found by Duncan Booth:
# It may be possible that bin_depends needs extension modules,
# so the loadpath must be extended by our python path.
loadpath = loadpath + ';' + ';'.join(pypath)

Can you try to change the last line into

loadpath = ';'.join(pypath) + ';' + loadpath

and report if this helps?

Thanks,

Thomas

 

[Luc Saffre]

I would consider this as a bug (in both py2exe and Installer) since
they decide to pick some cygwin tcl/tk dll's who happen to hang around
while the Python runtime won't...

This is part of the code (in build_exe.py).

def find_dependend_dlls(self, use_runw, dlls, pypath, dll_excludes):
import py2exe_util
sysdir = py2exe_util.get_sysdir()
windir = py2exe_util.get_windir()
# This is the tail of the path windows uses when looking for dlls
# XXX On Windows NT, the SYSTEM directory is also searched
exedir = os.path.dirname(sys.executable)
syspath = os.environ['PATH']
loadpath = ';'.join([exedir, sysdir, windir, syspath])

# Found by Duncan Booth:
# It may be possible that bin_depends needs extension modules,
# so the loadpath must be extended by our python path.
loadpath = loadpath + ';' + ';'.join(pypath)

Can you try to change the last line into

loadpath = ';'.join(pypath) + ';' + loadpath

and report if this helps?

Yes, this fixed the bug. Here is my test report:

1. I reproduced the bug by running a build with cygwin's tcltk package
installed: again, some part of cygwin had been grabbed. This time
however the behaviour was that the test.exe got stucked completely,
needing to be killed using TaskManager. This difference may come because
I upgraded meanwhile my copy of Cygwin.

2. I modified the line in build_exe.py as described.

3. Now the produced .exe runs well in both cases (tcltk installed during
build or not)

Sorry for the long delay before testing this.

Luc