Step by step: Compiling extensions with MS Visual C++ Toolkit 2003 - msvccompiler-patch.txt (0/1)

Discussion in 'Python' started by Martin Bless, Dec 15, 2004.

  1. Martin Bless

    Martin Bless Guest

    The good news:

    Along with Python-2.4 comes really good news to Windows users. Yes,
    you now CAN build extension modules yourself using the SAME C++
    compiler and linker Python is built with itself. Everything you need
    is available at no costs (except download hassle and installation
    time). Once your system is set up properly its just a matter of
    running 'python setup.py build'. No longer waiting for someone else to
    build binaries and a relief to those who kindly used to provide those
    binaries.


    The bad news:

    It's not easy to get there! Gathering information and packages and
    finding out about all the little steps and stumbling blocks may be
    very time consuming.


    Relax:

    Here's (much of) what you need to know. With this document I'm trying
    to provide the information I would have needed. It's sort of a step by
    step instruction and contains some details. Goal: Compile Python-2.4
    extensions with the freely available "Microsoft Visual C++ Toolkit
    2003". This document DOES NOT address the issue of compiling Python
    itself.

    The example chosen to build is the "MySQLdb" module with its
    _mysql.pyd extension.

    Go and check out Mike Fletcher's page first at

    http://www.vrplumber.com/programming/mstoolkit/

    It's his work (and the need to get a working MySQLdb for Python-2.4 of
    course ;-) that motivated me to start and spend some hours. Without
    Mike's page I wouldn't even have tried. Thank you so much! Now I'm
    prepared.

    Two things first - not to forget:

    (1) In contrast to what Mike writes I had to use a different registry
    key (see step 9)

    (2) I don't now what people mean when talking about "msvcr71.lib".
    There's no such file on my machine. BUT there IS a "msvcrt.lib" a
    directory "C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib". I
    guess the ".\Vc7\." indicates we are talking of the expected lib.
    Correct? I don't know.

    Attention - long lines:
    Watch out for long lines that may have gotten split in this posting!

    See if you've got something already. These are important directories
    we're looking for. When "echo %ProgramFiles%" gives "C:\Programme"
    there should be:

    C:\Programme\Microsoft Platform SDK for Windows XP SP2\Include
    C:\Programme\Microsoft Platform SDK for Windows XP SP2\Lib
    C:\Programme\Microsoft Visual C++ Toolkit 2003\bin
    C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib
    C:\Programme\Microsoft.NET\SDK\v1.1\include
    C:\Programme\Microsoft.NET\SDK\v1.1\Lib


    Step By Step
    ============

    1. Set up a Windows machine
    1.1 Install Windows XP (Prof. or Home doesn't matter)
    1.2 Apply Service Pack 2 (WinXP SP2)
    1.3 Perform a Windows update at
    http://windowsupdate.microsoft.com
    to get the latest patches.


    2. Get some tools

    This is optional - you don't have to. But it may be instructive and
    helpful. And - I would say - fun.

    2.1 Get "Windows Resource Kits"
    Brings some useful tools like "CMD Prompt Here" to the windows
    explorer context menu or "robocopy.exe" - a copy tools really worth to
    look at. Seems MS developers need good tools too ...

    http://www.microsoft.com/windows/reskits/default.asp

    The website says:
    """
    Windows Server 2003 Resource Kit Tools
    The Resource Kit Tools are a set of software tools for network
    administrators, developers, and power users to manage TCP/IP,
    networks, the registry, security, remote administration,
    configuration, batch files, and many other areas of the Windows Server
    2003 operating system. To install the tools, download and run
    Rktools.exe.
    """
    (rktools.exe, 12.337.752 bytes).



    2.2 Add some Unix to Windows. Goto

    http://unxutils.sourceforge.net/

    and get UnxUtils.zip (4.125 kB)
    """
    Here are some ports of common GNU utilities to native Win32. In this
    context, native means the executables do only depend on the Microsoft
    C-runtime (msvcrt.dll) and not an emulation layer like that provided
    by Cygwin tools.
    """
    For example, this gives you "diff.exe" and "patch.exe".



    3. Get and install the the .NET Framework 1.1.

    It's available at no costs from the MicroSoft website. Use Google to
    find the download page.

    3.1 Install the ".NET Framework 1.1 runtime" first (dotnetfx.exe,
    24.277.024 bytes)

    3.2 Install the .NET Framework 1.1 SDK" second. (setup(sdk1.1).exe,
    117.062.664 bytes).

    Attention, observation:

    This step should indeed create the badly needed

    C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib

    folder. It didn't on my machine until I first completely deinstalled
    on existing installation of .NET Framework 1.1 runtime AND .NET
    Framework 1.1 SDK and then resumed with steps 3.1 and 3.2. Only
    Redmond will know why - and I doubt it.

    3.3 Perform a Windows update at
    http://windowsupdate.microsoft.com
    to get the latest patches of the .NET Framework.



    4. Platform SDK

    Get and install the platform SDK "Windows XP SP2 SDK" (about
    266.000.00 bytes) from
    http://www.microsoft.com/msdownload/platformsdk/sdkupdate/XPSP2FULLInstall.htm

    You better download the whole stuff (several .CAB-files) first instead
    of trying to install online. This may be too risky.



    5. Get and install free "Visual C++ Toolkit 2003"

    This is the C++ compiler and linker. Use Google to find the "Microsoft
    VisualC++ Toolkit 2003" download page and get "VCToolkitSetup.exe"
    (32.948.392 bytes).

    The website says:
    """
    Overview
    The Microsoft Visual C++ Toolkit 2003 includes the core tools
    developers need to compile and link C++-based applications for Windows
    and the .NET Common Language

    Runtime:
    Microsoft C/C++ Optimizing Compiler and Linker. These are the same
    compiler and linker that ship with Visual Studio .NET 2003
    Professional!

    C Runtime Library and the C++ Standard Library, including the Standard
    Template Library. These are the same static-link libraries included
    with Visual Studio.

    Microsoft .NET Framework Common Language Runtime. Visual C++ can
    optionally build applications that target the Common LanguageRuntime
    (CLR).

    Sample code. The toolkit includes four samples designed to showcase
    the powerful new features of the 2003 version, including new
    optimization capabilities, features to improve code-security and
    robustness, enhanced ISO C++ standards support, and the ability to use
    the .NET Framework library and target the CLR.
    """


    6. Be sure that all components are up to date and do a Windows update
    at
    http://windowsupdate.microsoft.com


    7. Install Python-2.4 to C:\Python24


    8. Modify the Python distutils

    Make a backup of C:\Python24\Lib\distutils\msvccompiler.py and apply
    the patches as described in

    http://www.vrplumber.com/programming/mstoolkit/

    to this file. You can do this by hand or use "patch.exe" which you've
    gotten if you did follow step 2.2. There should be a file
    "msvccompiler-patch.txt" appended to this message.

    patch.exe msvccompiler.py msvccompiler-patch.txt

    should succeed with no error.



    9. Verify registry key of platform SDK

    Start "regedit.exe" from the commandline and check for the registry
    key of the platform SKD.

    ON MY MACHINE and in my case it is different from the one listed on
    http://www.vrplumber.com/programming/mstoolkit/

    It is
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05

    and it is not
    ....\63DADB24-DC99-45EB-A748-EC93AB8A7497.

    Find out about the correct key in your case and adjust the crucial
    line 133 in msvccompiler.py:

    freeSDK=r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05"

    If the key is wrong, you may encounter something like:

    """
    C:\> python setup.py build
    running build
    running build_ext
    error: Python was built with version 7.1 of Visual Studio, and
    extensions need to be built with the same version of the ompiler, but
    it isn't installed.
    """



    10. Create a script file "pathsetup.bat" hat helps you set up the
    necessary paths. We don't need to add the Python24 lib and include
    paths as the distutils will take care for this automatically.

    ++++++++++++++++
    REM START of pathsetup.bat, mb, 2004-12-12

    REM add Python-2.4 to path
    set Path=C:\Python24;%Path%

    REM add the compiler path
    set Path=C:\Programme\Microsoft Visual C++ Toolkit 2003\bin;%Path%

    REM setup include paths
    set Include=%ProgramFiles%\Microsoft Platform SDK for Windows XP
    SP2\Include;%Include%
    set Include=%ProgramFiles%\Microsoft Visual C++ Toolkit
    2003\include;%Include%

    REM setup Lib paths
    set Lib=C:\Programme\Microsoft Platform SDK for Windows XP
    SP2\Lib;%Lib%
    set Lib=C:\Programme\Microsoft Visual C++ Toolkit 2003\lib;%Lib%
    set Lib=C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib;%xxLib%

    REM END of pathsetup.bat
    ++++++++++++++++



    11. Source distribution of Python module

    Get and unpack the source distribution of a module you want to build.
    In our case get "MySQL-python-1.1.7.tar.gz" from
    http://sourceforge.net/projects/mysql-python or

    http://sourceforge.net/project/showfiles.php?group_id=22307&package_id=34790
    und unpack to ...\somehwere\MySQL-python-1.1.7


    11.1 Read README and setup.py

    Is there a README in "MySQL-python-1.1.7\"? Yes, there is! It tells
    you to get additional header and library files from the MySQL website.
    Looking for version 4.x at

    http://www.mysql.com/downloads

    we'll follow version 4.x and be taken to

    http://dev.mysql.com/downloads/mysql/4.0.html

    Locate "Windows downloads (platform notes)" on that page and download
    the binary distribution ("Windows (x86) Without installer (unzip in
    C:\) 4.0.22 20.8M"). The file is "mysql-4.0.22-win-noinstall.zip"
    (21.346 kB).

    Since we're not interested in running MySQL but only in building the
    extension we unzip into ...\MySQL-python-1.1.7\mysql-4.0.22-win



    11.2
    Make sure the MySQL header and library files are accessible by the C++
    compiler and linker. The location is hardcoded in 'setup.py' of the
    MySQL-python-1.1.7 module.

    Search for "win32" in 'setup.py' and change references to
    'C:\myslq\...' to '.\mysql-4.0.22-win

    ### >>>>>> schnipp >>>>>>>
    # setup.py of MySQL-python-1.1.7

    elif sys.platform == "win32": # Ugh
    include_dirs = [r'.\mysql-4.0.22-win\include']
    library_dirs = [r'.\mysql-4.0.22-win\lib\opt']
    extra_objects = [r'.\mysql-4.0.22-win\lib\opt\mysqlclient.lib']

    ### <<<<<< schnapp <<<<<<<



    12.
    Relax. Take a breath. We're ready to go :)

    12.1
    Open a command window.

    12.2
    Run "pathsetup.bat" to setup the environment.

    12.3 Change directory and step into ...\MySQL-python-1.1.7, that is,
    into the root dir of the module you want to build - the dir that
    contains the "setup.py" file.

    12.4 Start building the module including the extension using distutils
    with the command:

    python setup.py build


    13.
    There aren't fatal errors? Congratulations, you made it! Why not
    install then:

    python setup.py install

    should do the trick. (Yes, I had two or three warnings. But they don't
    seem to be important).


    14. Earn what you deserve

    Start Python and import:

    Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on
    win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import MySQLdb
    >>> <---------- imported! no error here!


    Yes, it does import :) And it does work :) And _mysql.pyd is similar
    in size to those versions I had before (about 185 kB) :)

    Life goes on. A different time another thread I'm going to ask about
    the warnings I got with building MySQLdb ...


    mb - Martin
    Martin Bless, Dec 15, 2004
    #1
    1. Advertising

  2. Martin Bless

    Fuzzyman Guest

    Cool.... it's nice to see these working. I followed Mike Fletchers
    instructions and they worked fine for me.

    I used mingw to get the patch tool.
    I did the core SDK download by doing a 'full download' (all 13 CAB
    files and extraction bat separately) and a local install. This is about
    a 350 odd meg download and some people have had issues with the MS
    installer.

    I've used it to build PyCrypto 2.0 successfully.

    Anyway - thanks for the info.

    Regards,
    Fuzzy
    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Fuzzyman, Dec 15, 2004
    #2
    1. Advertising

  3. Re: Step by step: Compiling extensions with MS Visual C++ Toolkit2003 - msvccompiler-patch.txt (0/1)

    Martin Bless wrote:
    ....

    >Two things first - not to forget:
    >
    >(1) In contrast to what Mike writes I had to use a different registry
    >key (see step 9)
    >
    >

    Which is expected (even noted on the page), particularly if you have a
    different version of the SDKs. The keys in the patch were extracted
    from an English Win2K Platform SDK. Don't know of any *good* way to
    figure out the keys in a version-agnostic manner. Suggestions welcome...

    >(2) I don't now what people mean when talking about "msvcr71.lib".
    >There's no such file on my machine. BUT there IS a "msvcrt.lib" a
    >directory "C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib". I
    >guess the ".\Vc7\." indicates we are talking of the expected lib.
    >Correct? I don't know.
    >
    >

    This was sloppy on my part. msvcr71.dll is the DLL used for Visual
    Studio 7.1, we want the .lib for that version of the DLL, but as you
    note, it is actually named msvcrt.lib (same as the one for Visual Studio
    6.0).

    >Get and install the platform SDK "Windows XP SP2 SDK" (about
    >
    >

    ....

    >It is
    >HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05
    >
    >and it is not
    >...\63DADB24-DC99-45EB-A748-EC93AB8A7497.
    >
    >Find out about the correct key in your case and adjust the crucial
    >line 133 in msvccompiler.py:
    >
    >freeSDK=r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05"
    >
    >

    ....
    Will have to figure out how to make this work across SDK versions somehow.

    Thanks for the report,
    Mike

    ________________________________________________
    Mike C. Fletcher
    Designer, VR Plumber, Coder
    http://www.vrplumber.com
    http://blog.vrplumber.com
    Mike C. Fletcher, Dec 15, 2004
    #3
  4. Martin Bless

    miki Guest

    Hello Martin,

    Martin Bless wrote:
    [Something very long ...]

    Why don't just use MingW32?
    Download it from
    http://prdownloads.sf.net/mingw/MinGW-3.1.0-1.exe?download (14.8MB) and
    install.

    Write your extension module and setup.py and then run:
    python setup.py build --compiler=mingw32
    That's all! You'll have a working .pyd ready to rock.

    Miki
    miki, Dec 16, 2004
    #4
  5. Martin Bless

    Fuzzyman Guest

    Fuzzyman, Dec 16, 2004
    #5
  6. Martin Bless

    Guest

    Having done steps 1 to 10, I tried building Numeric-23.6. And got the
    following error-msg:

    F:\install\Numeric-23.6>python setup.py build
    running build
    running build_py
    running build_ext
    building '_numpy' extension
    D:\Programme\Microsoft Visual C++ Toolkit 2003\bin\cl.exe /c /nologo
    /Ox /MD /W3
    /GX /DNDEBUG -I/usr/include/atlas -IInclude -IPackages\FFT\Include
    -IPackages\R
    NG\Include -ID:\Python24\include -ID:\Python24\PC /TcSrc\_numpymodule.c
    /Fobuild
    \temp.win32-2.4\Release\Src\_numpymodule.obj
    _numpymodule.c
    d:\Python24\include\pyconfig.h(30) : fatal error C1083: Cannot open
    include file
    : 'io.h': No such file or directory
    error: command '"D:\Programme\Microsoft Visual C++ Toolkit
    2003\bin\cl.exe"' fai
    led with exit status 2

    why?
    Regards
    Wolfgang
    , Dec 17, 2004
    #6
  7. Martin Bless

    Fuzzyman Guest

    wrote:
    > Having done steps 1 to 10, I tried building Numeric-23.6. And got the
    > following error-msg:
    >
    > F:\install\Numeric-23.6>python setup.py build
    > running build
    > running build_py
    > running build_ext
    > building '_numpy' extension
    > D:\Programme\Microsoft Visual C++ Toolkit 2003\bin\cl.exe /c /nologo
    > /Ox /MD /W3
    > /GX /DNDEBUG -I/usr/include/atlas -IInclude -IPackages\FFT\Include
    > -IPackages\R
    > NG\Include -ID:\Python24\include -ID:\Python24\PC

    /TcSrc\_numpymodule.c
    > /Fobuild
    > \temp.win32-2.4\Release\Src\_numpymodule.obj
    > _numpymodule.c
    > d:\Python24\include\pyconfig.h(30) : fatal error C1083: Cannot open
    > include file
    > : 'io.h': No such file or directory
    > error: command '"D:\Programme\Microsoft Visual C++ Toolkit
    > 2003\bin\cl.exe"' fai
    > led with exit status 2
    >
    > why?
    > Regards
    > Wolfgang


    If you look at the instructions on :
    http://www.vrplumber.com/programming/mstoolkit/

    It mentions that you probably need the .NET 2.0beta runtime and SDK.

    Regards,

    Fuzzy
    http://www.voidspace.org.uk/atlantibots/pythonutils.html
    Fuzzyman, Dec 17, 2004
    #7
  8. Martin Bless

    Guest

    ....
    > d:\Python24\include\pyconfig.h(30) : fatal error C1083: Cannot open
    > include file
    > : 'io.h': No such file or directory
    > error: command '"D:\Programme\Microsoft Visual C++ Toolkit
    > 2003\bin\cl.exe"' fai
    > led with exit status 2
    >
    > why?


    Under :
    HKEY_LOCAL_MACHINE\Software\Microsoft\VisualStudio
    \7.1\VC\VC_OBJECTS_PLATFORM_INFO\Win32\Directories

    You need to add the strings:
    "Include Dirs"(path to toolkit \include; path to platform sdk \include)
    "Library Dirs"(path to toolkit \lib; path to platform sdk \lib)
    "Path Dirs" (path to toolkit \bin; path to platform sdk \bin)
    , Dec 22, 2004
    #8
  9. Martin Bless

    Guest

    Is it possible to compile extension modules on windows platforms using
    MS Visual C++ Express 2005, or Visual Studio 8? If so, how would one
    modify the instructions posted in this thread? [assume I also have
    Toolkit 2003 installed, if necessary, for msvcr71.lib]

    Marcus


    Mike C. Fletcher wrote:
    > Martin Bless wrote:
    > ...
    >
    > >Two things first - not to forget:
    > >
    > >(1) In contrast to what Mike writes I had to use a different

    registry
    > >key (see step 9)
    > >
    > >

    > Which is expected (even noted on the page), particularly if you have

    a
    > different version of the SDKs. The keys in the patch were extracted
    > from an English Win2K Platform SDK. Don't know of any *good* way to
    > figure out the keys in a version-agnostic manner. Suggestions

    welcome...
    >
    > >(2) I don't now what people mean when talking about "msvcr71.lib".
    > >There's no such file on my machine. BUT there IS a "msvcrt.lib" a
    > >directory "C:\Programme\Microsoft Visual Studio .NET 2003\Vc7\lib".

    I
    > >guess the ".\Vc7\." indicates we are talking of the expected lib.
    > >Correct? I don't know.
    > >
    > >

    > This was sloppy on my part. msvcr71.dll is the DLL used for Visual
    > Studio 7.1, we want the .lib for that version of the DLL, but as you
    > note, it is actually named msvcrt.lib (same as the one for Visual

    Studio
    > 6.0).
    >
    > >Get and install the platform SDK "Windows XP SP2 SDK" (about
    > >
    > >

    > ...
    >
    > >It is

    >
    >HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05
    > >
    > >and it is not
    > >...\63DADB24-DC99-45EB-A748-EC93AB8A7497.
    > >
    > >Find out about the correct key in your case and adjust the crucial
    > >line 133 in msvccompiler.py:
    > >

    >
    >freeSDK=r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05"
    > >
    > >

    > ...
    > Will have to figure out how to make this work across SDK versions

    somehow.
    >
    > Thanks for the report,
    > Mike
    >
    > ________________________________________________
    > Mike C. Fletcher
    > Designer, VR Plumber, Coder
    > http://www.vrplumber.com
    > http://blog.vrplumber.com
    , Mar 18, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jody Burns
    Replies:
    8
    Views:
    401
    Grumman
    Dec 11, 2004
  2. Lucas Raab

    msvccompiler issues

    Lucas Raab, May 1, 2005, in forum: Python
    Replies:
    0
    Views:
    334
    Lucas Raab
    May 1, 2005
  3. Andreas
    Replies:
    0
    Views:
    441
    Andreas
    Dec 31, 2005
  4. Replies:
    3
    Views:
    420
  5. v4vijayakumar

    Free Microsoft Visual C++ Toolkit 2003

    v4vijayakumar, May 3, 2006, in forum: C Programming
    Replies:
    26
    Views:
    4,825
    Flash Gordon
    May 9, 2006
Loading...

Share This Page