How to build extensions on Windows?

[Kevin D. Smith]

I've written a simple Python extension for UNIX, but I need to get it
working on Windows now. I'm having some difficulties figuring out how
to do this. I've seen web pages that say that MS Visual Studio is
required, and other that say that's not true, that MinGW will work.
Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

 

[Lawrence Oluyede]

Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

I managed to set up a compilation toolchain in Windows following that
tutorial so what's your problem?

I installed MS .NET 1.1 and its SDK, the Platform SDK for Windows 2003
sever and the mstoolkit (you have to borrow it from somewhere because
it's not available anymore)
Then I hacked distutils and all worked well.

The only issue is to find the MS Toolkit 2003...

 

[Sybren Stuvel]

Kevin D Smith enlightened us with:

I've written a simple Python extension for UNIX, but I need to get
it working on Windows now. I'm having some difficulties figuring
out how to do this.

I had to do the same, and I didn't get much result. My solution:
install Cygwin, use the Python that comes with that, and use gcc just
like you're used to. Works like a charm, but the compiled extension is
incompatible with the regular Windows Pythons from python.org and
ActiveState.

Sybren

 

[Kevin D. Smith]

I managed to set up a compilation toolchain in Windows following that
tutorial so what's your problem?

I installed MS .NET 1.1 and its SDK, the Platform SDK for Windows 2003
sever and the mstoolkit (you have to borrow it from somewhere because
it's not available anymore)
Then I hacked distutils and all worked well.

The only issue is to find the MS Toolkit 2003...

So in other words, what you're saying is that the only issue I have
left is the exact issue that I described in my initial post that you
claimed isn't a problem... Great! I guess I'l get right down to work
compiling that extension now.

 

[Lawrence Oluyede]

So in other words, what you're saying is that the only issue I have
left is the exact issue that I described in my initial post that you
claimed isn't a problem... Great! I guess I'l get right down to work
compiling that extension now.

What I mean is that you have to find a way to get the toolkit. I don't
think MS will sue you if you borrow the compiler from a friend or
"download" it. Otherwise you can try with MingW I guess...

 

[Jason]

I've written a simple Python extension for UNIX, but I need to get it
working on Windows now. I'm having some difficulties figuring out how
to do this. I've seen web pages that say that MS Visual Studio is
required, and other that say that's not true, that MinGW will work.
Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

I don't know about MinGW, but you can get the Microsoft compilers by
installing Visual C++ 2005 Express. I'm guessing the old toolkit is
deprecated. While you must register each Visual Studio Express module
that you download, I don't think the actual command-line tools are
encumbered.

Why not try it out and let us know how it goes?

(Visual Studio 2005 Express:
http://msdn.microsoft.com/vstudio/express/)

--Jason

 

[Filip Wasilewski]

I've written a simple Python extension for UNIX, but I need to get it
working on Windows now. I'm having some difficulties figuring out how
to do this. I've seen web pages that say that MS Visual Studio is
required, and other that say that's not true, that MinGW will work.
Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

There is an easy way to build Python extensions on Windows with MinGW
and it works fine for me. Just follow these steps:


1. Get MinGW gcc and/or g++, preferably via MinGW installer from [1].
You may have to restart your computer or manually edit PATH system
environment variable to include MinGW's bin directory (default is
c:\mingw\bin). Then check if it is there by typing `path` in the cmd
window.

2. Get pexports-0.42h.zip from [2] and extract pexports.exe file

3. Prepare MinGW compatible .a library file
pexports.exe c:\WINDOWS\system32\python24.dll > python24.def
c:\mingw\bin\dlltool.exe --dllname python24.dll --def python24.def
--output-lib libpython24.a

4. Place the new libpython24.a file in the Python's libs directory (but
not in the Lib dir), default is c:\python24\libs

5. Build your extension by executing your setup script with
`--compiler=mingw32` parameter.
python setup.py build --compiler=mingw32

Additionally you may wish to put a distutils.cfg file in the
c:\python\lib\distutils dir containing following entries:
[build]
compiler = mingw32

This will tell the distutils script to use MinGW compiler by default
when executing `python setup.py build` command.

best,
fw

[1] http://sourceforge.net/projects/mingw/
[2] http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip

 

[Lawrence Oluyede]

I don't know about MinGW, but you can get the Microsoft compilers by
installing Visual C++ 2005 Express. I'm guessing the old toolkit is
deprecated. While you must register each Visual Studio Express module
that you download, I don't think the actual command-line tools are
encumbered.

Why not try it out and let us know how it goes?

(Visual Studio 2005 Express:
http://msdn.microsoft.com/vstudio/express/)

Because you can't use it to distribute extensions. Python is compiled
against the 2003 version and distutils will simply don't let you compile
with the 2005 version. If you manage to hack distutils to use the 2005
anyway it won't work for sure outside your home box.

 

[casevh]

1. Get MinGW gcc and/or g++, preferably via MinGW installer from [1].

You may have to restart your computer or manually edit PATH system
environment variable to include MinGW's bin directory (default is
c:\mingw\bin). Then check if it is there by typing `path` in the cmd
window.

1a. [Optional] Install MSYS, also from [1]. If your extension relies on
another C library, this may make it easier to compile that library.

2. Get pexports-0.42h.zip from [2] and extract pexports.exe file

3. Prepare MinGW compatible .a library file
pexports.exe c:\WINDOWS\system32\python24.dll > python24.def
c:\mingw\bin\dlltool.exe --dllname python24.dll --def python24.def
--output-lib libpython24.a

4. Place the new libpython24.a file in the Python's libs directory (but
not in the Lib dir), default is c:\python24\libs

I believe steps 2 through 4 are only required for Python 2.3 or
earlier. I have not needed to do those steps for Python 2.4 and 2.5.

5. Build your extension by executing your setup script with
`--compiler=mingw32` parameter.
python setup.py build --compiler=mingw32

Additionally you may wish to put a distutils.cfg file in the
c:\python\lib\distutils dir containing following entries:
[build]
compiler = mingw32

This will tell the distutils script to use MinGW compiler by default
when executing `python setup.py build` command.

best,
fw

[1] http://sourceforge.net/projects/mingw/
[2] http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip

 

[Jarek Zgoda]

Filip Wasilewski napisa³(a):

There is an easy way to build Python extensions on Windows with MinGW
and it works fine for me. Just follow these steps:

It was brougt to my attention that mingw-compiled extensions for Python
2.4 use other malloc() that Python 2.4, is it true? Can this have any
impact on stability?

 

[Filip Wasilewski]

Filip Wasilewski napisa³(a):


It was brougt to my attention that mingw-compiled extensions for Python
2.4 use other malloc() that Python 2.4, is it true? Can this have any
impact on stability?

I have inspected the dll dependencies for python24.dll (ActiveState
build, but I think this is also true for regular one) and my custom
extensions built with MinGW and both of them use malloc() from
msvcr71.dll. I think that as long as the extensions are linked against
proper version of msvcr (7.1) there should not be such surprises.

Other thing is that Python uses it's own memory allocator [1] and to
avoid problems with memory corruption one should not try to allocate
memory with PyMem_Malloc() and free it with free() and so forth.

cheers,
fw

[1] http://www.python.org/doc/2.4.3/api/memoryOverview.html

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Then there is Mike Fletcher's web page
(http://www.vrplumber.com/programming/mstoolkit/) that describes in
detail how to build extensions, but most of the links to external
software are no longer valid. I think it's safe to say that I am
completely lost, as there appears to be no authoritative, up-to-date
description on how to make this work.

If all else fails, buy a copy of Visual Studio 2003. They are available
fairly cheap at Ebay.

Regards,
Martin

 

[Kevin D. Smith]

I don't know about MinGW, but you can get the Microsoft compilers by
installing Visual C++ 2005 Express. I'm guessing the old toolkit is
deprecated. While you must register each Visual Studio Express module
that you download, I don't think the actual command-line tools are
encumbered.

Why not try it out and let us know how it goes?

(Visual Studio 2005 Express:
http://msdn.microsoft.com/vstudio/express/)

This almost worked (at least, it appears to). I got the module to
build and install using VS 2005. It works fine if I run python from
the directory where I built the extension. However, if you go to any
other directory, run python, and try to import the module, I get the
following error:

ImportError: dynamic module does not define init function (initsasSQL)

Why would it work from one directory, but not another?

 

[Fredrik Lundh]

This almost worked (at least, it appears to). I got the module to
build and install using VS 2005. It works fine if I run python from
the directory where I built the extension. However, if you go to any
other directory, run python, and try to import the module, I get the
following error:

ImportError: dynamic module does not define init function (initsasSQL)

Why would it work from one directory, but not another?

do you perhaps have a different "sassql.dll" file in your Python path?

(Python's standard import mechanism looks for PYD, DLL, PY, PYW, and
PYC, in that order.)

</F>

 

[John Machin]

do you perhaps have a different "sassql.dll" file in your Python path?

(Python's standard import mechanism looks for PYD, DLL, PY, PYW, and
PYC, in that order.)

Any support for the radical notion of the error message giving the full
path of the file that it's complaining about? If so, I'll lob in an
enhancement request in the morning ...

Cheers,
John

 

[Fredrik Lundh]

Any support for the radical notion of the error message giving the full
path of the file that it's complaining about? If so, I'll lob in an
enhancement request in the morning ...

there's always "python -vv", of course.

</F>

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

Any support for the radical notion of the error message giving the full
path of the file that it's complaining about? If so, I'll lob in an
enhancement request in the morning ...

A patch would be appreciated much more so.

Regards,
Martin

 

[John Machin]

there's always "python -vv", of course.

Thank you; I wasn't aware of that. Given that (as at 2.4.3) searching
the .chm docs for "interpreter", "command", and "option" don't yield
any pointers to the interpreter command line options, and given that
python -h gives no clue:
"""
-u : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
see man page for details on internal buffering relating to
'-u'
-v : verbose (trace import statements) (also PYTHONVERBOSE=x)
-V : print the Python version number and exit
-W arg : warning control (arg is action:message:category:module:lineno)
"""

could you please explain "of course"? I'm not sure how the general
populace could have known, apart from vague recollections of some Unix
utilities using a -v, -vv, -vvv etc convention.

TIA,
John

 

[Fuzzyman]

If all else fails, buy a copy of Visual Studio 2003. They are available
fairly cheap at Ebay.

For some value of 'cheap'. They seem to go for about £100 on ebay in
the UK. :-(

More interestingly, someone reported on Python-dev recently a speed
improvement of around 30% (from memory) by compiling with VC 8. I know
the grumble (almost certainly correctly) about Microsoft's 'odd'
interpretation of the C standards in VC 8, but it looks like there are
major benefits to switching...

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[John Machin]

A patch would be appreciated much more so.

Hi Martin, I do hope you don't regret opening Pandora's box

Does the following look about right?
I was somewhat bemused by the limit of 200 bytes on the module name in
the error message-- I woild have thought about 20 was more appropriate.
Sorry but I can't test this (Windows box, don't have whatever version
of C compiler is necessary to compile Python).

Cheers,
John
8<---
--- Python/importdl.c.orig 2006-09-14 09:53:59.984375000 +1000
+++ Python/importdl.c 2006-09-14 09:55:53.625000000 +1000
@@ -44,8 +44,8 @@
return NULL;
if (p == NULL) {
PyErr_Format(PyExc_ImportError,
- "dynamic module does not define init function
(init%.200s)",
- shortname);
+ "dynamic module (%s) does not define init function
(init%.200s)",
+ pathname, shortname);
return NULL;
}
oldcontext = _Py_PackageContext;
8<---