Are *.pyd's universal?

[Bakes]

Can I use a pyd compiled on linux in a Windows distribution?

Or must I recompile it for windows users?

 

[Diez B. Roggisch]

Can I use a pyd compiled on linux in a Windows distribution?

Or must I recompile it for windows users?

Yes, you must.

Diez

 

[Jerry Hill]

Can I use a pyd compiled on linux in a Windows distribution?
No.

Or must I recompile it for windows users?

Yes.

 

[Lawrence D'Oliveiro]

On Linux and several other Unices the suffix is .so and not .pyd.

Why is that? Or conversely, why isn't it .dll under Windows?

 

[Philip Semanchuk]

.so is the common suffix of shared libraries on Linux. IIRC Python
extensions have .pyd on Mac OS X.

I've never seen a .pyd under OS X (although that doesn't mean they
don't exist).

The Python extensions I've written in C compile to a .so under OS X.

Cheers
P

 

[Robert Kern]

I've never seen a .pyd under OS X (although that doesn't mean they don't
exist).

The Python extensions I've written in C compile to a .so under OS X.

This is true of all Python extensions on OS X. .pyd is only used on Windows.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Lawrence D'Oliveiro]

On Windows it used to be .dll, too.
The suffix was changed to avoid issues with other dlls and name clashes.

What clashes? How come other OSes don't seem prone to the same problems?

 

[Carl Banks]

What clashes? How come other OSes don't seem prone to the same problems?

Yeah, because who ever heard of one OS having a problem that another
OS didn't?

Windows searches for DLLs on the executable path, which always
includes the current directory. So if you have a module called
system32.dll in your currently directory, you could be in big trouble.

Unix doesn't automatically search the current dir, doesn't use the
executable search path (libraries have their own search path, which is
not used when loading shared libs by hand), and system libraries on
Unix conventionally are prefixed by lib. So name collisions are rare,
but even if if you have a module called libc.so in your current
directory it will not conflict with /lib/libc.so.


Carl Banks

 

[Dave Angel]

Yeah, because who ever heard of one OS having a problem that another
OS didn't?

Windows searches for DLLs on the executable path, which always
includes the current directory. So if you have a module called
system32.dll in your currently directory, you could be in big trouble.

Unix doesn't automatically search the current dir, doesn't use the
executable search path (libraries have their own search path, which is
not used when loading shared libs by hand), and system libraries on
Unix conventionally are prefixed by lib. So name collisions are rare,
but even if if you have a module called libc.so in your current
directory it will not conflict with /lib/libc.so.


Carl Banks

I don't believe changing the extension modifies the search order for
LoadLibrary(). However, it does make a name collision much less likely,
which is a "goodthing." And I'd bet that more than 50% of DLL's on a
typical machine have some other extension.

As for search order, I believe it's a bit different than the one used by
CMD to search for an executable. I do not think it includes the current
directory, but instead includes the executable directory (which is
frequently where Python.exe is located).

And I'm guessing that CPython searches down sys.path, and when it finds
the module, gives a full path to LoadLibrary(), in which case the DLL
search path is moot.

DaveA

 

[Carl Banks]

And I'm guessing that CPython searches down sys.path, and when it finds
the module, gives a full path to LoadLibrary(), in which case the DLL
search path is moot.

It's not Python that's the issue. The issue is that if you have a
module with a .dll extension, other programs could accidentally try to
load that module instead of the intended dll, if the module is in the
current directory or system path. Modules will sometimes find
themselves on the path in Windows, so the fact that Windows performs a
library search on the path is quite significant.


Carl Banks

 

[Lawrence D'Oliveiro]

In message <6e603d9c-2be0-449c-9c3c-

It's not Python that's the issue. The issue is that if you have a
module with a .dll extension, other programs could accidentally try to
load that module instead of the intended dll, if the module is in the
current directory or system path. Modules will sometimes find
themselves on the path in Windows, so the fact that Windows performs a
library search on the path is quite significant.

Why is it only Windows is prone to this problem?

 

[Albert Hopkins]

Modules will sometimes find

Why is it only Windows is prone to this problem?

I think as someone pointed out earlier, in Unix-like operating systems,
a "regular" library's file name starts with "lib", e.g. libcrypt.so. So
this would not conflict with Python's crypt.so. But in Windows, they
would both be named crypt.dll, for example (I'm halfway guessing since I
don't have/use Windows).

 

[Lawrence D'Oliveiro]

On Sat, 2009-10-31 at 21:32 +1300, Lawrence D'Oliveiro wrote:



I think as someone pointed out earlier, in Unix-like operating systems,
a "regular" library's file name starts with "lib", e.g. libcrypt.so. So
this would not conflict with Python's crypt.so.

I just checked my Debian installation:

ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\* -print | wc -l
2950
ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l
4708

So 63% of the shareable libraries on my system have names NOT beginning with lib.

Any better theories?

 

[Carl Banks]

In message <6e603d9c-2be0-449c-9c3c-



Why is it only Windows is prone to this problem?

 

[Carl Banks]

In message <6e603d9c-2be0-449c-9c3c-



Why is it only Windows is prone to this problem?

Unix doesn't search the executable path or current directory when
loading libraries, which if you would have seen if you hadn't ignored
my original reply.


Carl Banks

 

[Albert Hopkins]

I just checked my Debian installation:

ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\*
-print | wc -l
2950
ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l
4708

So 63% of the shareable libraries on my system have names NOT
beginning with lib.

Any better theories?

Those are likely not system (sharable) libraries (e.g. libcrypt). These
are probably plugins loaded by a specific program, for example PAM
modules, ImageMagick plugins, python modules, etc. so since they are
not in your library path the do not stand getting accidentally loaded
(e.g. when a binary is linked against libcrypt.so). These libraries are
loaded directly by the program using their exact path name and dlopen().

The issue with windows they were saying is that Windows will load the
library with that name if it is in your current directory. So if you
happen to have a python library called CRYPT.DLL and you are in that
directory and try to run a program that loads CRYPT.DLL then you will be
loading the python module instead of the Windows one (I have no idea if
Windows has a CRYPT.DLL).

OTOH this doesn't happen in Linux because a) programs wanting the
system's crypt library are looking for libcrypt.so and b) Linux doesn't
look in your current directory (by default) for libraries.

 

[Carl Banks]

OTOH this doesn't happen in Linux because a) programs wanting the
system's crypt library are looking for libcrypt.so and b) Linux doesn't
look in your current directory (by default) for libraries.

One other thing is that linux binaries are usually linked against a
versioned .so file, so a random command would refernce "libc.so.6"
rather than "libc.so".

I don't think that's necessarily the case for all Unix-like OSes,
though.


Carl Banks

 

[Steven D'Aprano]

I question your assumption. "DLL hell" is merely a platform-specific name
for a particular variety of a broader class of problems, namely
dependency conflicts and the bootstrapping problem. It's not even
specific to software -- bootstrapping problems are well known in many
fields of endeavour: before you can make the tools to make things, you
need to make the tools to make the tools...

It is far from true to say that Windows is the only system subject to
this problem.

http://en.wikipedia.org/wiki/Dependency_hell

Python, like most (all?) programming languages, has it's own version of
dependency hell, namely circular imports: module A depends on module B,
which depends on module C, which depends on module A...

Python also has a second "DLL Hell", as many newbies discover: shadowing
standard library modules. Create a module (say) "random.py" in the
current directory, and then try to import the standard library random.
This is a form of DLL Hell, because Python uses standard library modules
dynamically, importing them at runtime.

One solution to dependency conflicts and circular dependencies is to
avoid dependencies altogether. At the cost of disk space and memory, use
static linking and give up the benefits of dynamic libraries. Platforms
that encourage static linking avoid the DLL conflicts, but at the cost of
increased memory and storage usage, and static applications need to be
upgraded individually instead of merely upgrading the shared library.


But it would conflict with a Python module libcrypt.so if the PYTHONPATH
and the OS's shared library path coincided.

I just checked my Debian installation:

ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\*
-print | wc -l 2950
ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l 4708

So 63% of the shareable libraries on my system have names NOT beginning
with lib.

53% on my system. This just goes to show that even Linux developers often
don't bother with the conventions for their platforms.