How does one get from "ImportError: DLL load failed:..." to a culprit.dll and symbol?

[Chris Cormie]

Hi,

I've been Googling around on a moderately common Windows Python problem:
a mismatch between the symbols a python extension thinks are available
and the contents of the associated DLL. Python users running into this
problem are greeted with:

import <somemodule>
"ImportError: DLL load failed: The specified procedure could not be found."

That's it: no mention of which dll nor which symbol is causing the
problem. Both these pieces of information are helpful/important in
resolving this class of problem: how does one get Python to dump this
debugging data?

(>python -v -d
doesn't help, BTW)

Regards,
Chris

 

[John Machin]

Hi,

I've been Googling around on a moderately common Windows Python problem:
a mismatch between the symbols a python extension thinks are available
and the contents of the associated DLL. Python users running into this
problem are greeted with:

import <somemodule>
"ImportError: DLL load failed: The specified procedure could not be found.."

That's it: no mention of which dll nor which symbol is causing the
problem. Both these pieces of information are helpful/important in
resolving this class of problem: how does one get Python to dump this
debugging data?

(>python -v -d
doesn't help, BTW)

Use -vv ... you need to know exactly which of possibly many
incarnations of somemodule.pyd Python is trying to import from.

Then get a copy of the Dependency Walker from http://www.dependencywalker.com/

Open c:\the\failing\somemodule.pyd with the DW. Look at grumble
messages in the bottom pane. Note: you often get 1 or 2 warning
messages with a pyd that's not having problems. Scroll through all the
modules in the middle pane, looking for grumbles.

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

By the way, I presume that you are aware that extensions on Windows
are tied to a particular version of Python ... you would have got a
different error message if you had a version mismatch (I think). If
your problem is not obvious, come back here with:
-- what version of Python you are running
-- what DLLs you find in the middle pane whose names match
(1) MSVC*.DLL
(2) PYTHON??.DLL

You may need to answer questions about the extension (e.g. compiler/
linker options used) -- is it your extension or a 3rd party's?

HTH,
John

 

[Chris Cormie]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person
Your technique works well and it does provide the information and it is
a (roundabout) solution to this class of problems: thank you very much.
It doesn't answer the original question so I'll restate it:

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails. At that point it
has the name of the dll and the problem symbol to hand and yet strangely
only gives an opaque error message. How does one get Python to print out
the faulty DLL and symbol?

(Even if it means a debug build of Python and adding the printf
yourself, I just want to know where we stand on this issue! )

Best Regards,
Chris

 

[John Machin]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person

It wasn't apparent, and still isn't.

Your technique works well and it does provide the information and it is
a (roundabout) solution to this class of problems:  thank you very much..
It doesn't answer the original question so I'll restate it:

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails. At that point it
has the name of the dll and the problem symbol to hand and yet strangely
only gives an opaque error message. How does one get Python to print out
the faulty DLL and symbol?

If you know all that, then the answer is to submit a patch, isn't it??

(Even if it means a debug build of Python and adding the printf
yourself,

printf? Maybe you shouldn't submit a patch after all.

 

[Gabriel Genellina]

En Mon, 23 Feb 2009 10:41:20 -0200, Chris Cormie

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person
Your technique works well and it does provide the information and it is
a (roundabout) solution to this class of problems: thank you very much.
It doesn't answer the original question so I'll restate it:

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails. At that point it
has the name of the dll and the problem symbol to hand and yet strangely
only gives an opaque error message. How does one get Python to print out
the faulty DLL and symbol?

You can't, because it isn't Python who's trying to load the symbol - the
*only* symbol that Python attempts to import itself is "initFOO" from
FOO.pyd, and when that fails you get an explicit message.
The error you see must come from the extension itself, and propagates into
Python as an ImportError.

 

[Mark Hammond]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person

LOL sounds right!

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails.

It is the LoadLibrary that is failing; it should be obvious that if it
was a simple GetProcAddress that was failing, Python would simply throw
an exception rather than displaying the ugly dialog box you see.

The problem is that some *other* DLL in the chain of dependencies is
failing to load; please re-adjust your perceptions of your own knowledge
and re-read John's initial response - if you follow his instructions
that should all become quite obvious.

Mark

 

[Chris Cormie]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person

LOL sounds right!

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails.

It is the LoadLibrary that is failing; it should be obvious that if it
was a simple GetProcAddress that was failing, Python would simply throw
an exception rather than displaying the ugly dialog box you see.

I'm talking about the whole class of errors when loading an extension,
the LoadLibary succeeds but the GetProcAddress fails. Not one specific
error, a class of errors and *by definition* from the original question
LoadLibrary succeeds. That's the point: at some point in this common
scenario we have the path to the dll and a symbol we are resolving, and
the symbol name, yet for whatever reason the GetProcAddress lookup
fails, resulting in a surprisingly uninformative errors message.

Regards,
Chris.

 

[Chris Cormie]

En Mon, 23 Feb 2009 10:41:20 -0200, Chris Cormie

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person
Your technique works well and it does provide the information and it
is a (roundabout) solution to this class of problems: thank you very
much. It doesn't answer the original question so I'll restate it:

How do you get *Python* to tell you the dll and the problem symbol?
Not external tools, Python. Python at a low level is calling
LoadLibrary and GetProcAddress to resolve symbols and the call fails.
At that point it has the name of the dll and the problem symbol to
hand and yet strangely only gives an opaque error message. How does
one get Python to print out the faulty DLL and symbol?

You can't, because it isn't Python who's trying to load the symbol - the
*only* symbol that Python attempts to import itself is "initFOO" from
FOO.pyd, and when that fails you get an explicit message.
The error you see must come from the extension itself, and propagates
into Python as an ImportError.

Right -- now we are getting somewhere! That makes a lot of sense.

What I was expecting to see next was an explicit GetProcAddress() call
in the extension code that was failing in initFOO() or elsewhere in the
extension code. But there are no such calls.

I think what might have been making this confusing is that the root of
my example problem is link time.

Let me explain by giving the specific example case I have to hand.
I orginally got on to this class of opaque runtime errors caused by
missing symbols through building the pycURL extension on MinGW. As of
Python 2.6, Python links against the new msvcr90.dll C Runtime. MinGW
maintains a set of .a files for system DLLs including the C Runtime.
When building pycURL naturally I want to link against libmsvcr90.a so
Python 2.6 and pycURL are using the same C Rutime. Problem was, MinGW's
.a was wrong: it exports symbols that msvcr90.dll does not in fact
possess. Building pycURL with the faulty libmsvcr90.a causes the opaque
error message: "ImportError: DLL load failed"

Now I expected this meant an explict LoadLibrary() GetProcAddress()
sequence was failing in the extension but perhaps that's not what
happens at all?
Chris

 

[Chris Cormie]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person

It wasn't apparent, and still isn't.

Yikes, enough with the rudeness people. I'm asking a reasonable question
in a good faith.

 

[Chris Cormie]

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along

LOL, I am that person

LOL sounds right!

How do you get *Python* to tell you the dll and the problem symbol? Not
external tools, Python. Python at a low level is calling LoadLibrary and
GetProcAddress to resolve symbols and the call fails.

It is the LoadLibrary that is failing; it should be obvious that if it
was a simple GetProcAddress that was failing, Python would simply throw
an exception rather than displaying the ugly dialog box you see.

The problem is that some *other* DLL in the chain of dependencies is
failing to load; please re-adjust your perceptions of your own knowledge
and re-read John's initial response - if you follow his instructions
that should all become quite obvious.

Mark

I think an example would help rather than me jabbering on about
LoadLibary/GetProcAddress .

It really is a case of Python loads module x depending on DLL y that is
missing symbol z, yet the error message is just "load x failed" when in
theory it could be "load x failed: y doesn't have z".

However I think I am confusing people (and myself!) by assuming it's
always as explicit as a LoadLibary/GetProcAddress sequence that is
failing when people see this error.

Let me explain by giving the specific example case I have to hand.
I originally got on to this class of opaque runtime errors caused by
missing symbols through building the pycURL extension on MinGW. As of
Python 2.6, Python links against the new msvcr90.dll C Runtime. MinGW
maintains a set of .a files for system DLLs including the C Runtime.
When building pycURL naturally I want to link against libmsvcr90.a so
Python 2.6 and pycURL are using the same C Runtime. Problem was, MinGW's
.a was wrong: it exports symbols that msvcr90.dll does not in fact
possess. Building pycURL with the faulty libmsvcr90.a causes the opaque
error message: "ImportError: DLL load failed"

I think Gabriel's got the part of the answer: she said "it isn't Python
who's trying to load the symbol - the *only* symbol that Python attempts
to import itself is "initFOO" from FOO.pyd"

So my next question is: given the above missing-symbol problem, is the
system able/willing to report the bad symbol reference in the .pyd when
Python attempts to import initFOO?

Regards,
Chris.

 

[John Machin]

On 23/02/2009 11:41 PM, Chris Cormie wrote:

If that not-very-technical description [all I've ever needed] doesn't
help, you'll need to read the DW help file (HTFF1K) or wait till
someone who knows what they are doing comes along
LOL, I am that person

LOL sounds right!

It is the LoadLibrary that is failing; it should be obvious that if it
was a simple GetProcAddress that was failing, Python would simply throw
an exception rather than displaying the ugly dialog box you see.

The problem is that some *other* DLL in the chain of dependencies is
failing to load; please re-adjust your perceptions of your own knowledge
and re-read John's initial response - if you follow his instructions
that should all become quite obvious.

Mark

I think an example would help rather than me jabbering on about
LoadLibary/GetProcAddress .

It really is a case of Python loads module x depending on DLL y that is
missing symbol z, yet the error message is just "load x failed" when in
theory it could be "load x failed: y doesn't have z".

However I think I am confusing people (and myself!) by assuming it's

Don't assume; read the [open] source code; see below.

always as explicit as a LoadLibary/GetProcAddress sequence that is
failing when people see this error.

Let me explain by giving the specific example case I have to hand.
I originally got on to this class of opaque runtime errors caused by
missing symbols through building the pycURL extension on MinGW. As of
Python 2.6, Python links against the new msvcr90.dll C Runtime. MinGW
maintains a set of .a files for system DLLs including the C Runtime.
When building pycURL naturally I want to link against libmsvcr90.a so
Python 2.6 and pycURL are using the same C Runtime. Problem was, MinGW's
  .a was wrong: it exports symbols that msvcr90.dll does not in fact
possess. Building pycURL with the faulty libmsvcr90.a causes the opaque
error message: "ImportError: DLL load failed"

I think Gabriel's got the part of the answer: she said  "it isn't Python
who's trying to load the symbol - the *only* symbol that Python attempts
to import itself is "initFOO" from FOO.pyd"

So my next question is: given the above missing-symbol problem, is the
system able/willing to report the bad symbol reference in the .pyd when
Python attempts to import initFOO?

My reading of the Python 2.6.1 version of dynload_win.c tells me that:

1. It uses LoadLibraryEx() to attempt to load the pyd given its full
pathname

2. If that fails, it gets the Windows error code using GetLastError()

3. It gets the Windows message text using FormatMessage()

4. It assembles the message as "DLL load failed: " plus the Windows
message text (less any trailing "\r\n").

Note that if the pyd load succeeds, it rummages in memory to find the
pythonxx.dll used by the pyd so that it can be checked for
consistency. Then and only then it uses GetProcAddress() to get the
address of the initFOO function.

Do you know a way of getting more info out of Windows than GetLastError
()?

 

[Chris Cormie]

My reading of the Python 2.6.1 version of dynload_win.c tells me that:

1. It uses LoadLibraryEx() to attempt to load the pyd given its full
pathname

2. If that fails, it gets the Windows error code using GetLastError()

3. It gets the Windows message text using FormatMessage()

4. It assembles the message as "DLL load failed: " plus the Windows
message text (less any trailing "\r\n").

Note that if the pyd load succeeds, it rummages in memory to find the
pythonxx.dll used by the pyd so that it can be checked for
consistency. Then and only then it uses GetProcAddress() to get the
address of the initFOO function.

Do you know a way of getting more info out of Windows than GetLastError
()?

Thank you for your help, it is much appreciated:

In summary:

Q) How can one get Python to tell you which symbol is causing a problem
when it loads an extension with a bad reference to a symbol in a DLL it
uses?

A) There is no such way: you *must* use external tools: When a symbol is
missing form a DLL that a Python extension depends on, the failure will
be picked up in Python immediately on attempting to load the extension:
the extension code is never reached and there is no specific attempt
made in Python to resolve the problem symbol. further, The normal
platform APIs lack the ability to determine the cause of the load failure.

> Do you know a way of getting more info out of Windows than GetLastError
> ()?

No, but presumably the ability to determine in code that a DLL has a bad
reference and which symbol is bad exists because the depends program is
able to do it. Of course it's a separate question as to whether Python
wants to load itself down with complex platform specific code for a
small corner case.

Best Regards,
Chris.