Are *.pyd's universal?

Discussion in 'Python' started by Bakes, Oct 29, 2009.

  1. Bakes

    Bakes Guest

    Can I use a pyd compiled on linux in a Windows distribution?

    Or must I recompile it for windows users?
     
    Bakes, Oct 29, 2009
    #1
    1. Advertising

  2. Bakes wrote:

    > Can I use a pyd compiled on linux in a Windows distribution?
    >
    > Or must I recompile it for windows users?


    Yes, you must.

    Diez
     
    Diez B. Roggisch, Oct 29, 2009
    #2
    1. Advertising

  3. Bakes

    Jerry Hill Guest

    On Thu, Oct 29, 2009 at 12:44 PM, Bakes <> wrote:
    > Can I use a pyd compiled on linux in a Windows distribution?


    No.

    > Or must I recompile it for windows users?


    Yes.
     
    Jerry Hill, Oct 29, 2009
    #3
  4. In message <>, Christian
    Heimes wrote:

    > On Linux and several other Unices the suffix is .so and not .pyd.


    Why is that? Or conversely, why isn't it .dll under Windows?
     
    Lawrence D'Oliveiro, Oct 30, 2009
    #4
  5. On Oct 29, 2009, at 8:41 PM, Christian Heimes wrote:

    > Lawrence D'Oliveiro schrieb:
    >> In message <>,
    >> Christian
    >> Heimes wrote:
    >>
    >>> On Linux and several other Unices the suffix is .so and not .pyd.

    >>
    >> Why is that? Or conversely, why isn't it .dll under Windows?

    >
    > .so is the common suffix of shared libraries on Linux. IIRC Python
    > extensions have .pyd on Mac OS X.


    I've never seen a .pyd under OS X (although that doesn't mean they
    don't exist).

    The Python extensions I've written in C compile to a .so under OS X.

    Cheers
    P
     
    Philip Semanchuk, Oct 30, 2009
    #5
  6. Bakes

    Robert Kern Guest

    Philip Semanchuk wrote:
    >
    > On Oct 29, 2009, at 8:41 PM, Christian Heimes wrote:
    >
    >> Lawrence D'Oliveiro schrieb:
    >>> In message <>,
    >>> Christian
    >>> Heimes wrote:
    >>>
    >>>> On Linux and several other Unices the suffix is .so and not .pyd.
    >>>
    >>> Why is that? Or conversely, why isn't it .dll under Windows?

    >>
    >> .so is the common suffix of shared libraries on Linux. IIRC Python
    >> extensions have .pyd on Mac OS X.

    >
    > I've never seen a .pyd under OS X (although that doesn't mean they don't
    > exist).
    >
    > The Python extensions I've written in C compile to a .so under OS X.


    This is true of all Python extensions on OS X. .pyd is only used on Windows.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Oct 30, 2009
    #6
  7. In message <>, Christian
    Heimes wrote:

    > Lawrence D'Oliveiro schrieb:
    >
    >> In message <>,
    >> Christian Heimes wrote:
    >>
    >>> On Linux and several other Unices the suffix is .so and not .pyd.

    >>
    >> Why is that? Or conversely, why isn't it .dll under Windows?

    >
    > On Windows it used to be .dll, too.
    > The suffix was changed to avoid issues with other dlls and name clashes.


    What clashes? How come other OSes don't seem prone to the same problems?
     
    Lawrence D'Oliveiro, Oct 30, 2009
    #7
  8. Bakes

    Carl Banks Guest

    On Oct 29, 9:10 pm, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    > In message <>, Christian
    >
    > Heimes wrote:
    > > Lawrence D'Oliveiro schrieb:

    >
    > >> In message <>,
    > >> Christian Heimes wrote:

    >
    > >>> On Linux and several other Unices the suffix is .so and not .pyd.

    >
    > >> Why is that? Or conversely, why isn't it .dll under Windows?

    >
    > > On Windows it used to be .dll, too.
    > > The suffix was changed to avoid issues with other dlls and name clashes..

    >
    > What clashes? How come other OSes don't seem prone to the same problems?


    Yeah, because who ever heard of one OS having a problem that another
    OS didn't?

    Windows searches for DLLs on the executable path, which always
    includes the current directory. So if you have a module called
    system32.dll in your currently directory, you could be in big trouble.

    Unix doesn't automatically search the current dir, doesn't use the
    executable search path (libraries have their own search path, which is
    not used when loading shared libs by hand), and system libraries on
    Unix conventionally are prefixed by lib. So name collisions are rare,
    but even if if you have a module called libc.so in your current
    directory it will not conflict with /lib/libc.so.


    Carl Banks
     
    Carl Banks, Oct 30, 2009
    #8
  9. Bakes

    Dave Angel Guest

    Carl Banks wrote:
    > On Oct 29, 9:10 pm, Lawrence D'Oliveiro <l...@geek-
    > central.gen.new_zealand> wrote:
    >
    >> In message <>, Christian
    >>
    >> Heimes wrote:
    >>
    >>> Lawrence D'Oliveiro schrieb:
    >>>
    >>>> In message <>,
    >>>> Christian Heimes wrote:
    >>>>
    >>>>> On Linux and several other Unices the suffix is .so and not .pyd.
    >>>>>
    >>>> Why is that? Or conversely, why isn't it .dll under Windows?
    >>>>
    >>> On Windows it used to be .dll, too.
    >>> The suffix was changed to avoid issues with other dlls and name clashes.
    >>>

    >> What clashes? How come other OSes don't seem prone to the same problems?
    >>

    >
    > Yeah, because who ever heard of one OS having a problem that another
    > OS didn't?
    >
    > Windows searches for DLLs on the executable path, which always
    > includes the current directory. So if you have a module called
    > system32.dll in your currently directory, you could be in big trouble.
    >
    > Unix doesn't automatically search the current dir, doesn't use the
    > executable search path (libraries have their own search path, which is
    > not used when loading shared libs by hand), and system libraries on
    > Unix conventionally are prefixed by lib. So name collisions are rare,
    > but even if if you have a module called libc.so in your current
    > directory it will not conflict with /lib/libc.so.
    >
    >
    > Carl Banks
    >
    >

    I don't believe changing the extension modifies the search order for
    LoadLibrary(). However, it does make a name collision much less likely,
    which is a "goodthing." And I'd bet that more than 50% of DLL's on a
    typical machine have some other extension.

    As for search order, I believe it's a bit different than the one used by
    CMD to search for an executable. I do not think it includes the current
    directory, but instead includes the executable directory (which is
    frequently where Python.exe is located).

    And I'm guessing that CPython searches down sys.path, and when it finds
    the module, gives a full path to LoadLibrary(), in which case the DLL
    search path is moot.

    DaveA
     
    Dave Angel, Oct 30, 2009
    #9
  10. Bakes

    Carl Banks Guest

    On Oct 30, 8:43 am, Dave Angel <> wrote:
    > And I'm guessing that CPython searches down sys.path, and when it finds
    > the module, gives a full path to LoadLibrary(), in which case the DLL
    > search path is moot.


    It's not Python that's the issue. The issue is that if you have a
    module with a .dll extension, other programs could accidentally try to
    load that module instead of the intended dll, if the module is in the
    current directory or system path. Modules will sometimes find
    themselves on the path in Windows, so the fact that Windows performs a
    library search on the path is quite significant.


    Carl Banks
     
    Carl Banks, Oct 30, 2009
    #10
  11. In message <6e603d9c-2be0-449c-9c3c-
    >, Carl Banks wrote:

    > It's not Python that's the issue. The issue is that if you have a
    > module with a .dll extension, other programs could accidentally try to
    > load that module instead of the intended dll, if the module is in the
    > current directory or system path. Modules will sometimes find
    > themselves on the path in Windows, so the fact that Windows performs a
    > library search on the path is quite significant.


    Why is it only Windows is prone to this problem?
     
    Lawrence D'Oliveiro, Oct 31, 2009
    #11
  12. On Sat, 2009-10-31 at 21:32 +1300, Lawrence D'Oliveiro wrote:
    > Modules will sometimes find
    > > themselves on the path in Windows, so the fact that Windows performs

    > a
    > > library search on the path is quite significant.

    >
    > Why is it only Windows is prone to this problem?


    I think as someone pointed out earlier, in Unix-like operating systems,
    a "regular" library's file name starts with "lib", e.g. libcrypt.so. So
    this would not conflict with Python's crypt.so. But in Windows, they
    would both be named crypt.dll, for example (I'm halfway guessing since I
    don't have/use Windows).
     
    Albert Hopkins, Oct 31, 2009
    #12
  13. In message <>, Albert Hopkins wrote:

    > On Sat, 2009-10-31 at 21:32 +1300, Lawrence D'Oliveiro wrote:
    >
    >In message <>, Carl Banks wrote:
    >
    >> Modules will sometimes find themselves on the path in Windows, so the
    >> fact that Windows performs a library search on the path is quite
    >> significant.
    >>
    >> Why is it only Windows is prone to this problem?

    >
    > I think as someone pointed out earlier, in Unix-like operating systems,
    > a "regular" library's file name starts with "lib", e.g. libcrypt.so. So
    > this would not conflict with Python's crypt.so.


    I just checked my Debian installation:

    ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\* -print | wc -l
    2950
    ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l
    4708

    So 63% of the shareable libraries on my system have names NOT beginning with lib.

    Any better theories?
     
    Lawrence D'Oliveiro, Oct 31, 2009
    #13
  14. Bakes

    Carl Banks Guest

    On Oct 31, 1:32 am, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    > In message <6e603d9c-2be0-449c-9c3c-
    >
    > >, Carl Banks wrote:
    > > It's not Python that's the issue.  The issue is that if you have a
    > > module with a .dll extension, other programs could accidentally try to
    > > load that module instead of the intended dll, if the module is in the
    > > current directory or system path.  Modules will sometimes find
    > > themselves on the path in Windows, so the fact that Windows performs a
    > > library search on the path is quite significant.

    >
    > Why is it only Windows is prone to this problem?
     
    Carl Banks, Oct 31, 2009
    #14
  15. Bakes

    Carl Banks Guest

    On Oct 31, 1:32 am, Lawrence D'Oliveiro <l...@geek-
    central.gen.new_zealand> wrote:
    > In message <6e603d9c-2be0-449c-9c3c-
    >
    > >, Carl Banks wrote:
    > > It's not Python that's the issue.  The issue is that if you have a
    > > module with a .dll extension, other programs could accidentally try to
    > > load that module instead of the intended dll, if the module is in the
    > > current directory or system path.  Modules will sometimes find
    > > themselves on the path in Windows, so the fact that Windows performs a
    > > library search on the path is quite significant.

    >
    > Why is it only Windows is prone to this problem?


    Unix doesn't search the executable path or current directory when
    loading libraries, which if you would have seen if you hadn't ignored
    my original reply.


    Carl Banks
     
    Carl Banks, Oct 31, 2009
    #15
  16. On Sat, 2009-10-31 at 23:58 +1300, Lawrence D'Oliveiro wrote:
    > I just checked my Debian installation:
    >
    > ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\*
    > -print | wc -l
    > 2950
    > ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l
    > 4708
    >
    > So 63% of the shareable libraries on my system have names NOT
    > beginning with lib.
    >
    > Any better theories?


    Those are likely not system (sharable) libraries (e.g. libcrypt). These
    are probably plugins loaded by a specific program, for example PAM
    modules, ImageMagick plugins, python modules, etc. so since they are
    not in your library path the do not stand getting accidentally loaded
    (e.g. when a binary is linked against libcrypt.so). These libraries are
    loaded directly by the program using their exact path name and dlopen().

    The issue with windows they were saying is that Windows will load the
    library with that name if it is in your current directory. So if you
    happen to have a python library called CRYPT.DLL and you are in that
    directory and try to run a program that loads CRYPT.DLL then you will be
    loading the python module instead of the Windows one (I have no idea if
    Windows has a CRYPT.DLL).

    OTOH this doesn't happen in Linux because a) programs wanting the
    system's crypt library are looking for libcrypt.so and b) Linux doesn't
    look in your current directory (by default) for libraries.
     
    Albert Hopkins, Oct 31, 2009
    #16
  17. Bakes

    Carl Banks Guest

    On Oct 31, 6:49 am, Albert Hopkins <> wrote:
    > OTOH this doesn't happen in Linux because a) programs wanting the
    > system's crypt library are looking for libcrypt.so and b) Linux doesn't
    > look in your current directory (by default) for libraries.


    One other thing is that linux binaries are usually linked against a
    versioned .so file, so a random command would refernce "libc.so.6"
    rather than "libc.so".

    I don't think that's necessarily the case for all Unix-like OSes,
    though.


    Carl Banks
     
    Carl Banks, Oct 31, 2009
    #17
  18. On Sat, 31 Oct 2009 23:58:33 +1300, Lawrence D'Oliveiro wrote:

    > In message <>, Albert
    > Hopkins wrote:
    >
    >> On Sat, 2009-10-31 at 21:32 +1300, Lawrence D'Oliveiro wrote:
    >>
    >>In message
    >><>, Carl
    >>Banks wrote:
    >>
    >>> Modules will sometimes find themselves on the path in Windows, so the
    >>> fact that Windows performs a library search on the path is quite
    >>> significant.
    >>>
    >>> Why is it only Windows is prone to this problem?


    I question your assumption. "DLL hell" is merely a platform-specific name
    for a particular variety of a broader class of problems, namely
    dependency conflicts and the bootstrapping problem. It's not even
    specific to software -- bootstrapping problems are well known in many
    fields of endeavour: before you can make the tools to make things, you
    need to make the tools to make the tools...

    It is far from true to say that Windows is the only system subject to
    this problem.

    http://en.wikipedia.org/wiki/Dependency_hell

    Python, like most (all?) programming languages, has it's own version of
    dependency hell, namely circular imports: module A depends on module B,
    which depends on module C, which depends on module A...

    Python also has a second "DLL Hell", as many newbies discover: shadowing
    standard library modules. Create a module (say) "random.py" in the
    current directory, and then try to import the standard library random.
    This is a form of DLL Hell, because Python uses standard library modules
    dynamically, importing them at runtime.

    One solution to dependency conflicts and circular dependencies is to
    avoid dependencies altogether. At the cost of disk space and memory, use
    static linking and give up the benefits of dynamic libraries. Platforms
    that encourage static linking avoid the DLL conflicts, but at the cost of
    increased memory and storage usage, and static applications need to be
    upgraded individually instead of merely upgrading the shared library.


    >> I think as someone pointed out earlier, in Unix-like operating systems,
    >> a "regular" library's file name starts with "lib", e.g. libcrypt.so.
    >> So this would not conflict with Python's crypt.so.


    But it would conflict with a Python module libcrypt.so if the PYTHONPATH
    and the OS's shared library path coincided.


    > I just checked my Debian installation:
    >
    > ldo@theon:~> find /lib /usr/lib -name \*.so -a -not -name lib\*
    > -print | wc -l 2950
    > ldo@theon:~> find /lib /usr/lib -name \*.so -print | wc -l 4708
    >
    > So 63% of the shareable libraries on my system have names NOT beginning
    > with lib.



    53% on my system. This just goes to show that even Linux developers often
    don't bother with the conventions for their platforms.



    --
    Steven
     
    Steven D'Aprano, Nov 1, 2009
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jerry Pine

    calldll.pyd for python 2.0

    Jerry Pine, Aug 9, 2003, in forum: Python
    Replies:
    0
    Views:
    369
    Jerry Pine
    Aug 9, 2003
  2. Liu Kun, SLC ICM (BJ)

    How to use a .pyd

    Liu Kun, SLC ICM (BJ), Nov 7, 2003, in forum: Python
    Replies:
    2
    Views:
    494
    Daniel Dittmar
    Nov 7, 2003
  3. Gary Herron
    Replies:
    0
    Views:
    395
    Gary Herron
    Dec 17, 2003
  4. Fuzzyman

    _ssl.pyd in py2exe ???

    Fuzzyman, Feb 21, 2004, in forum: Python
    Replies:
    5
    Views:
    553
    Fuzzyman
    Feb 23, 2004
  5. Jon Nicoll
    Replies:
    0
    Views:
    289
    Jon Nicoll
    Aug 12, 2004
Loading...

Share This Page