Problem building/using M2Crypto extension module

[Carl Waldbieser]

I tried to adapt the instructions for building the M2Crypto module
(http://sandbox.rulemaker.net/ngps/m2/INSTALL.html) to build a version
compatible with Python2.3, but I've had some mixed results. I actually got
everything to build and install, but when I try to import M2Crypto-- well,
here is a sample session:
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.

import M2Crypto
dir() ['M2Crypto', '__builtins__', '__doc__', '__name__', 'pywin']

import M2Crypto.Rand

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\Rand.py", line 7, in ?
import m2
File "C:\PYTHON23\Lib\site-packages\M2Crypto\m2.py", line 7, in ?
from __m2crypto import *
ImportError: DLL load failed: The system cannot find the file specified.
As you can see, the first time I try to import the module, it fails, but the
next time, it works. The same thing happens when I try to import a
sub-module underneath it.

Has anyone experienced any problems similar to this when building extension
modules? Is there some sort of common trap I might have fallen into?

The following is version info:
I am running Python 2.3.1 on Windows98. I built the extension from source,
using the mingw compiler 3.2.3. I built version 0.9-7c of the OpenSSL
library to use with M2Crypto. This is the first time I used SWIG 1.3.19 and
PExport 0.42h.

This is the first time I have actually tried to build a Python extension on
this PC, so the chances are probably pretty good that I have screwed
something up, but I am not sure how I would go about tracking it down.

If anyone else has successfully built and used this extension for Python2.3
on Windows, I would be glad to hear it, just to know it is possible. If
anyone has any advice or suggestions, I would also be glad to hear that,
too!

Thanks,
Carl Waldbieser

 

[Ng Pheng Siong]

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.

Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
somewhere?

I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD. No
change in build procedure.

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.
$ python2.3 alltests.py
......................................................................
----------------------------------------------------------------------
Ran 70 tests in 6.756s

OK


On Windows, I use Bloodshed DevC++, which is some version of mingw. Instead
of pydef, I use the free BC++'s impdef, but either should produce
equivalent .def files.

I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
then I noticed 2.3.2 release candidate, so I decided to wait a little.

Do you have Python 2.2.3, say? Are you able to build on this version?

 

[Carl Waldbieser]

There is a __m2crypto.pyd in my \Python23\Lib\site-packages\M2Crypto
directory. I don't have Python 2.2.3 installed, but I may eventually
re-install it just to see if I can get this working.

Is there some way I can inspect the output files at each step of the build
so as to detect an error in my process? Also, I do not recall using pydef,
and I am not sure what it is. I beilieve that I used a tool called pexport
to make the .def file for python.

If you get around to building it for Python2.3.x, I'd be glad to know how it
went. One thing I got hung up on for a long time was trying to get the
distutils to do the install. If I tried:

python setup.py install

I would get an error because I built the files using mingw and it would say
I needed to build it with Visual C++. Of course, I already had built it for
mingw with:

python setup.py build -cmingw

so I was a bit confused. There was no such flag for the install, as far as
I could tell. After reading through the help for a while, I found a flag
that let me skip the re-build that is implicit in the install. I didn't
think that should have caused a problem, though.

Carl Waldbieser

Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "C:\PYTHON23\Lib\site-packages\M2Crypto\__init__.py", line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.

Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
somewhere?

I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD. No
change in build procedure.

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.
$ python2.3 alltests.py
......................................................................
----------------------------------------------------------------------
Ran 70 tests in 6.756s

OK


On Windows, I use Bloodshed DevC++, which is some version of mingw. Instead
of pydef, I use the free BC++'s impdef, but either should produce
equivalent .def files.

I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
then I noticed 2.3.2 release candidate, so I decided to wait a little.

Do you have Python 2.2.3, say? Are you able to build on this version?


--
Ng Pheng Siong <[email protected]>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

 

[Ng Pheng Siong]

There is a __m2crypto.pyd in my \Python23\Lib\site-packages\M2Crypto
directory.

cd there, import the DLL into the interpreter toplevel:

$ ls -l __m2crypto.so
-rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.
Does it work?

Is there some way I can inspect the output files at each step of the build
so as to detect an error in my process? Also, I do not recall using pydef,
and I am not sure what it is. I beilieve that I used a tool called pexport
to make the .def file for python.

Sorry, typo, I meant pyexport.

I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
for use with mingw. Take a look and adjust paths (e.g., OpenSSL
include/library files, swig.exe) where necessary. Try it with "make -f
Makefile.mw".

 

[Ng Pheng Siong]

If you get around to building it for Python2.3.x, I'd be glad to know how it
went.

Turns out I do have Python 2.3 installed on one of my
rarely-booted-into-Windows computer.

Here's what I get:

C:\ngps\prog\ngps\m2\SWIG>make -f Makefile.mw
c:/pkg/swig/swig.exe -shadow -python _m2crypto.i
gcc -c -DTHREADING -g -Ic:/pkg/py23/include -Ic:/pkg/openssl/include -I. _m2cryp
to_wrap.c
[warnings deleted]
dllwrap --dllname __m2crypto.pyd --driver-name gcc \
--def _m2crypto.def -o __m2crypto.pyd _m2crypto_wrap.o \
-s --entry _DllMain@12 --target=i386-mingw32 c:/pkg/py23/libs/libpyt
hon23.a c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a
copy _m2crypto.py ..\M2Crypto
1 file(s) copied.
copy __m2crypto.pyd ..\M2Crypto
1 file(s) copied.

C:\ngps\prog\ngps\m2\SWIG>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

C:\ngps\prog\ngps\m2>python setup.py build -cmingw32
running build
running build_py
running build_ext
building '__m2crypto' extension
swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap.c
c:\pkg\swig\swig.exe -python -ISWIG -o SWIG/_m2crypto_wrap.c SWIG/_m2crypto.i
g:\pkg\dev-c++\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\ngps\prog\ngps\m2\SWI
G -Ic:\pkg\openssl/include -Ic:\pkg\py23\include -Ic:\pkg\py23\PC -c SWIG/_m2cry
pto_wrap.c -o build\temp.win32-2.3\Release\swig\_m2crypto_wrap.o -DTHREADING
[warnings deleted]
writing build\temp.win32-2.3\Release\swig\__m2crypto.def
g:\pkg\dev-c++\bin\dllwrap.exe -mno-cygwin -mdll -static --output-lib build\temp
..win32-2.3\Release\swig\lib__m2crypto.a --def build\temp.win32-2.3\Release\swig\
__m2crypto.def -s build\temp.win32-2.3\Release\swig\_m2crypto_wrap.o -Lc:\pkg\op
enssl\lib -Lc:\pkg\py23\libs -Lc:\pkg\py23\PCBuild -lssleay32 -llibeay32 -lpytho
n23 -o build\lib.win32-2.3\M2Crypto\__m2crypto.pyd

C:\ngps\prog\ngps\m2>python setup.py install --skip-build
running install
running install_lib
creating c:\pkg\py23\Lib\site-packages\M2Crypto
copying build\lib.win32-2.3\M2Crypto\ASN1.py -> c:\pkg\py23\Lib\site-packages\M2
Crypto
[blah blah blah]
byte-compiling c:\pkg\py23\Lib\site-packages\M2Crypto\__init__.py to __init__.py
c

C:\>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.['ASN1', 'BIO', 'DH', 'DSA', 'EVP', 'Err', 'PGP', 'RC4', 'RCS_id', 'RSA', 'Rand'
, 'SSL', 'X509', '__builtins__', '__doc__', '__file__', '__m2crypto', '__name__'
, '__path__', 'decrypt', 'encrypt', 'httpslib', 'm2', 'm2urllib', 'urllib2', 'ut
il']

.............................++++++++++++
..............++++++++++++

You need mingw versions of libpython23.a, libssl32.a and libeay32.a.

Makefile.mw says:

SSLLIB=c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a

Whereas, using setup.py, the command line constructed is "-lssleay32
-llibeay32", so you need libssleay32.a and liblibeay32.a. Somewhat
ridiculous looking, yes. ;-). This is because setup.py was written for
VC++; the relevant snippet looks like this:

if os.name == 'nt':
openssl_dir = 'c:\\pkg\\openssl'
include_dirs = [my_inc, openssl_dir + '/include']
library_dirs = [openssl_dir + '\\lib']
libraries = ['ssleay32', 'libeay32']
#libraries = ['ssleay32_bc', 'libeay32_bc']
extra_compile_args = [ "-DTHREADING" ]

So, alternatively, you can modify the "libraries" line to say "['ssl32',
'eay32']" (I think) to use the default library names assigned to OpenSSL
libraries by mingw. (On Unix they're called libssl.so and libcrypto.so.
Duh!)

Notice "['ssleay_bc', 'libeay_bc']". That's for Borland's free compiler
suite. I haven't tested this in a long while. Going forward, it will be
mingw only.

HTH.

 

[Carl Waldbieser]

cd there, import the DLL into the interpreter toplevel:

$ ls -l __m2crypto.so
-rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright", "credits" or "license" for more information.
Does it work?

No, it gives me an error each time. Guess there is something wrong with the
..pyd.

I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
for use with mingw. Take a look and adjust paths (e.g., OpenSSL
include/library files, swig.exe) where necessary. Try it with "make -f
Makefile.mw".

That seems to have worked. The .pyd is a little bigger. I copied it into
the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
run the tests, I get:

.................................................................EEEEEE
======================================================================
ERROR: test_cipher_mismatch (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
88, in test_cipher_mismatch
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

======================================================================
ERROR: test_cipher_ok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
114, in test_cipher_ok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

======================================================================
ERROR: test_no_such_cipher (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
101, in test_no_such_cipher
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

======================================================================
ERROR: test_server_simple (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
54, in test_server_simple
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

======================================================================
ERROR: test_tls1_nok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
65, in test_tls1_nok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

======================================================================
ERROR: test_tls1_ok (test_ssl_win.SSLWinClientTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl.py", line
77, in test_tls1_ok
pid = self.start_server(self.args)
File "C:\Downloads\m2crypto\m2crypto-0.12-snap1\tests\test_ssl_win.py",
line 41, in start_server
None, None, self.startupinfo)
error: (2, 'CreateProcess', 'The system cannot find the file specified.')

----------------------------------------------------------------------
Ran 70 tests in 2.530s

FAILED (errors=6)

So it looks like I'm not quite out of the woods yet, but these Python errors
at least seem a little more comprehensible to me. I am still not sure what
was going on with the library. Do you have any idea what I did wrong the
first time, or was it just a lucky guess? I would be interested in knowing
so I can try to avoid that kind of mistake in the future.

Thanks for your help!
Carl Waldbieser

 

[Ng Pheng Siong]

That seems to have worked. The .pyd is a little bigger. I copied it into
the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
run the tests, I get:

................................................................EEEEEE

These are ok; the errors are in test_ssl_win.py, which says:

Win32 version - requires Mark Hammond's Win32 extensions and openssl.exe
on your PATH.

So it looks like I'm not quite out of the woods yet, but these Python errors
at least seem a little more comprehensible to me. I am still not sure what
was going on with the library. Do you have any idea what I did wrong the
first time, or was it just a lucky guess?

I can't tell, really. As I posted in another message, building for Python
2.3 on Windows works for me.

Anyways, looks like you're good to go.

Cheers.

 

[Carl Waldbieser]

Yes, that was the problem-- openssl.exe was not in my PATH. Thanks!

Carl Waldbieser