When does a binary extension gets the file extension '.pyd' and whenis it '.so'

Discussion in 'Python' started by llothar, Apr 4, 2008.

  1. llothar

    llothar Guest

    On windows everything is '.pyd' but there seems to be two ways to get
    this on unix?
    Why and what is the rule?
     
    llothar, Apr 4, 2008
    #1
    1. Advertising

  2. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:

    > On windows everything is '.pyd' but there seems to be two ways to get
    > this on unix?


    If you attempt to import the module "spam" on Windows, Python looks for
    "spam.dll" and "spam.pyd" (in addition to "spam.py/spam.pyw/spam.pyc" etc)

    On most Unix platforms, Python looks for "spam.so" and "spammodule.so".

    You can check what suffixes a given Python version uses via the "imp"
    module:

    >>> import imp
    >>> imp.get_suffixes()


    To see *where* Python is looking as well, use the "-vv" flag:

    $ python -vv -c "import spam"
    ...
    # trying spam.so
    # trying spammodule.so
    # trying spam.py
    # trying spam.pyc
    ... etc

    (-vv prints loads of stuff, so you may want to use "grep" to filter out
    the stuff you're interested in:

    $ python -vv -c "import spam" 2>&1 | grep spam

    or, under Windows:

    > python -vv -c "import spam" 2> out.txt
    > findstr spam out.txt


    )

    > Why and what is the rule?


    If you want Python to be able to import your binary extension, make sure
    to use a name it looks for.

    </F>
     
    Fredrik Lundh, Apr 4, 2008
    #2
    1. Advertising

  3. llothar

    llothar Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Thanks, my question was not how can i make python to it find. I don't
    have a problem.

    My question was: Why does setup.py generated sometimes a pyd and
    sometimes a so file?

    There must be a rule behind this.

    Unforunately setup.py is not well documented. Here i mean i need a
    specification not a
    tutorial, because i want to know something not do something.

    Don't ask why i need to know: I need to know. And maybe somebody here
    can drop a
    line before i have to dig around in the source code.
     
    llothar, Apr 5, 2008
    #3
  4. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:

    > My question was: Why does setup.py generated sometimes a pyd and
    > sometimes a so file?


    setup.py picks an extension that happens to work on the platform you're
    running setup.py on. doing otherwise would be pretty pointless.

    </F>
     
    Fredrik Lundh, Apr 5, 2008
    #4
  5. llothar

    llothar Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    On 5 Apr., 15:48, Fredrik Lundh <> wrote:
    > llothar wrote:
    > > My question was: Why does setup.py generated sometimes a pyd and
    > > sometimes a so file?

    >
    > setup.py picks an extension that happens to work on the platform you're
    > running setup.py on. doing otherwise would be pretty pointless.
    >
    > </F>


    Unfortunately as pointless as the answers i got so far.


    Okay i try it one more time:

    I ship an application that compiles an python interpreter and
    extension on a remote system.
    It also needs to copy this created items around. So if i use setup.py
    to create an
    extension i need to know the file name of the generated file.

    Damned this is trivial and a fundamental question and it is not
    documented anywhere.

    I have a clue at the moment that it might be ".so" when python is
    compiled without shared library
    and ".pyd" otherwise (configure option --enable-shared) . But this is
    just a guess. Does anybody know?

    And by the way: I think this is a bug and should be fixed. If the
    platform does allow renaming the
    extension of a DLL (does HP/UX allow this?) it should always be
    ".pyd"
     
    llothar, Apr 5, 2008
    #5
  6. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:

    > I ship an application that compiles an python interpreter and
    > extension on a remote system.
    > It also needs to copy this created items around. So if i use setup.py
    > to create an
    > extension i need to know the file name of the generated file.


    so why not just ask setup.py to copy the files for you?

    you can either use "install" with the --home option to copy all the
    files to a given directory, or create a binary kit with "bdist" and
    unpack the resulting archive at the target location.

    > Unfortunately as pointless as the answers i got so far.


    well, cutting the "I'm not going to tell you why I need to know this"
    stuff might help people come up with solutions to your actual problem
    instead of posting stuff that's informative but misses the point.

    </F>
     
    Fredrik Lundh, Apr 5, 2008
    #6
  7. llothar

    Steve Holden Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:
    > On 5 Apr., 15:48, Fredrik Lundh <> wrote:
    >> llothar wrote:
    >>> My question was: Why does setup.py generated sometimes a pyd and
    >>> sometimes a so file?

    >> setup.py picks an extension that happens to work on the platform you're
    >> running setup.py on. doing otherwise would be pretty pointless.
    >>
    >> </F>

    >
    > Unfortunately as pointless as the answers i got so far.
    >

    Right, so you think people aren't trying to help you?

    Put your paranoia back in your pocket :)
    >
    > Okay i try it one more time:
    >
    > I ship an application that compiles an python interpreter and
    > extension on a remote system.
    > It also needs to copy this created items around. So if i use setup.py
    > to create an
    > extension i need to know the file name of the generated file.
    >
    > Damned this is trivial and a fundamental question and it is not
    > documented anywhere.
    >
    > I have a clue at the moment that it might be ".so" when python is
    > compiled without shared library
    > and ".pyd" otherwise (configure option --enable-shared) . But this is
    > just a guess. Does anybody know?
    >
    > And by the way: I think this is a bug and should be fixed. If the
    > platform does allow renaming the
    > extension of a DLL (does HP/UX allow this?) it should always be
    > ".pyd"


    You display your ignorance here. The ".pyd" extension is used on Windows
    as an alternative to ".dll", but both are recognized as shared
    libraries. Personally I'm not really sure why they even chose to use
    ".pyd", which is confusing to most Windows users. In UNIX/Linux
    environments ".so" is the standard extension for a shared library.

    To depart from the platform standard would be unhelpful and confusing to
    the majority of users. It's know use telling us what you think: tell us
    instead the compelling reasons why your opinion is correct. Opinions,
    after all, are so cheap that everyone can have one.

    There are ways to build distributions of Python extensions (modules or
    packages involving binary code from languages like C or C++), but you
    will want to understand a bit more about computing in general (and work
    on your social skills ;-) before you start to approach them.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Apr 5, 2008
    #7
  8. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Steve Holden wrote:

    > You display your ignorance here. The ".pyd" extension is used on Windows
    > as an alternative to ".dll", but both are recognized as shared
    > libraries. Personally I'm not really sure why they even chose to use
    > ".pyd", which is confusing to most Windows users. In UNIX/Linux
    > environments ".so" is the standard extension for a shared library.


    background:

    http://mail.python.org/pipermail/python-dev/1999-December/001456.html

    "Python cannot import every .dll file. Only .dll files that conform to
    the convention for Python extension modules can be imported. (The
    convention is that it must export an init<module> function.)

    On most other platforms, shared libraries must have a specific
    extension (e.g. .so on most Unix). Python allows you to drop such a
    file into any directory where is looks for modules, and it will then
    direct the dynamic load support to load that specific file.

    This seems logical -- Python extensions must live in directories that
    Python searches (Python must do its own search because the search
    order is significant).

    On Windows, Python uses the same strategy. The only modification is
    that it is allowed to give the file a different extension, namely
    ..pyd, to indicate that this really is a Python extension and not a
    regular DLL. This was mostly introduced because it is apparently
    common to have an existing DLL "foo.dll" and write a Python wrapper
    for it that is also called "foo". Clearly, two files foo.dll are too
    confusing, so we let you name the wrapper foo.pyd. But because the
    file format is essentially that of a DLL, we don't *require* this
    renaming; some ways of creating DLLs in the first place may make it
    difficult to do."

    </F>
     
    Fredrik Lundh, Apr 5, 2008
    #8
  9. llothar

    llothar Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    > Right, so you think people aren't trying to help you?

    I think they are not reading the question.

    > You display your ignorance here. The ".pyd" extension is used on Windows
    > as an alternative to ".dll", but both are recognized as shared
    > libraries. Personally I'm not really sure why they even chose to use
    > ".pyd", which is confusing to most Windows users.


    Here i agree. But having it's own identifiying extension has also some
    small benefits.

    > To depart from the platform standard would be unhelpful and confusing to
    > the majority of users. It's know use telling us what you think: tell us
    > instead the compelling reasons why your opinion is correct. Opinions,
    > after all, are so cheap that everyone can have one.


    Because i want a uniform solution. Either use "dllso" or use "pyd"
    but
    stay with one decision once made. At the moment when i build python on
    my ubuntu system without "--enable-shared" i get a pyd file created,
    if
    i use "--enable-shared" it is a so file.

    I don't know if this is a special case on Linux or the general on unix
    systems
    (i only have Linux, FreeBSD, NetBSD, OpenBSD, Solaris and MacOSX in
    mind).

    > There are ways to build distributions of Python extensions (modules or
    > packages involving binary code from languages like C or C++), but you
    > will want to understand a bit more about computing in general


    Believe me nobody needs to teach me anything about general programming
    anymore.

    > (and work on your social skills ;-)


    I don't think so. I asked a pretty simple question and as usual on
    usenet nobody
    read the question but answered to complete different topics. Answers
    on usenet are
    so cheap, everybody likes to give one - no matter if it is ontopic,
    right or wrong.
    And this does not really help.

    My question is simple and person who knows setup.py and distools would
    be able to
    give the answer in a small sentence if there is a strategy behind it
    and it's not
    only a bug.

    Unfortunately there is no python.core mailing list that i know so i
    ask here.
     
    llothar, Apr 5, 2008
    #9
  10. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:

    > I don't think so. I asked a pretty simple question and as usual on
    > usenet nobody read the question


    did *you* read your own question? it took you three posts before you
    mentioned what you were trying to do, and four posts before you bothered
    to mention that you're seeing this behaviour on Ubuntu.

    and for the record, Python doesn't look for PYD files on any of the Unix
    boxes I have convenient access to right now. what Ubuntu version are
    you using, what Python version do you have, and what does

    $ python -c "import imp; print imp.get_suffixes()"

    print on your machine?

    </F>
     
    Fredrik Lundh, Apr 5, 2008
    #10
  11. llothar

    Steve Holden Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar wrote:
    [...]
    > Unfortunately there is no python.core mailing list that i know so i
    > ask here.
    >

    Well your researches can't have been that extensive: the developers live
    on python-dev@pythonlorg, otherwise known as comp.land.python-dev. But
    you will need to ask your question rather more carefully there to het a
    meaningful answer. Good luck.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Apr 5, 2008
    #11
  12. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Fredrik Lundh wrote:

    > and for the record, Python doesn't look for PYD files on any of the Unix
    > boxes I have convenient access to right now. what Ubuntu version are
    > you using, what Python version do you have, and what does
    >
    > $ python -c "import imp; print imp.get_suffixes()"
    >
    > print on your machine?


    for reference, here's what I get on Ubuntu 7.10, with the standard
    Python interpreter (2.5.1):

    $ python -c "import imp; print imp.get_suffixes()"
    [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
    ('.pyc', 'rb', 2)]

    any Ubuntu gurus here that can sort this one out?

    </F>
     
    Fredrik Lundh, Apr 5, 2008
    #12
  13. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    > There must be a rule behind this.

    There are multiple rules behind this. Python normally uses
    the same extension for shared libraries as the operating
    system, as the operating system may refuse to load them if
    it doesn't. So it *can't* use .pyd on Unix, because that
    might not work (on some implementations of Unix).

    It can't use .dll on Windows, because that may (and did)
    conflict with existing DLLs, hence it uses .pyd.

    Depending on the system, an extension module may have one
    of these suffixes:
    - .so
    - module.so
    - .sl
    - module.sl
    - .exe
    - .EXE
    - module.exe
    - module.EXE
    - .pyd
    - _d.pyd

    Regards,
    Martin
     
    Martin v. Löwis, Apr 5, 2008
    #13
  14. llothar

    Steve Holden Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Fredrik Lundh wrote:
    > Fredrik Lundh wrote:
    >
    >> and for the record, Python doesn't look for PYD files on any of the Unix
    >> boxes I have convenient access to right now. what Ubuntu version are
    >> you using, what Python version do you have, and what does
    >>
    >> $ python -c "import imp; print imp.get_suffixes()"
    >>
    >> print on your machine?

    >
    > for reference, here's what I get on Ubuntu 7.10, with the standard
    > Python interpreter (2.5.1):
    >
    > $ python -c "import imp; print imp.get_suffixes()"
    > [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
    > ('.pyc', 'rb', 2)]
    >
    > any Ubuntu gurus here that can sort this one out?
    >

    I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
    team decide that you would be able to import extension module YYY either
    from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
    to answer that I have no idea at all.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Apr 6, 2008
    #14
  15. llothar

    Steve Holden Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Fredrik Lundh wrote:
    > Fredrik Lundh wrote:
    >
    >> and for the record, Python doesn't look for PYD files on any of the Unix
    >> boxes I have convenient access to right now. what Ubuntu version are
    >> you using, what Python version do you have, and what does
    >>
    >> $ python -c "import imp; print imp.get_suffixes()"
    >>
    >> print on your machine?

    >
    > for reference, here's what I get on Ubuntu 7.10, with the standard
    > Python interpreter (2.5.1):
    >
    > $ python -c "import imp; print imp.get_suffixes()"
    > [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
    > ('.pyc', 'rb', 2)]
    >
    > any Ubuntu gurus here that can sort this one out?
    >

    I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
    team decide that you would be able to import extension module YYY either
    from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
    to answer that I have no idea at all.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Apr 6, 2008
    #15
  16. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Steve Holden wrote:

    >> for reference, here's what I get on Ubuntu 7.10, with the standard
    >> Python interpreter (2.5.1):
    >>
    >> $ python -c "import imp; print imp.get_suffixes()"
    >> [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
    >> ('.pyc', 'rb', 2)]
    >>
    >> any Ubuntu gurus here that can sort this one out?
    >>

    > I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
    > team decide that you would be able to import extension module YYY either
    > from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
    > to answer that I have no idea at all.


    oh, the ".so" and "module.so" is standard Python behaviour (see my first
    post in this thread). what I cannot figure out is how "llothar" has
    managed to get setup.py to build extensions that an Ubuntu Python cannot
    load, without noticing.

    </F>
     
    Fredrik Lundh, Apr 6, 2008
    #16
  17. Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    llothar schreef:
    >> There are ways to build distributions of Python extensions (modules or
    >> packages involving binary code from languages like C or C++), but you
    >> will want to understand a bit more about computing in general

    >
    > Believe me nobody needs to teach me anything about general programming
    > anymore.


    "The people who are best at programming are the people who realize how
    small their brains are. They are humble." -- Edsger Dijkstra, 1972

    (http://www.codinghorror.com/blog/archives/000051.html,
    http://www.cs.utexas.edu/users/EWD/ewd03xx/EWD340.PDF)


    --
    The saddest aspect of life right now is that science gathers knowledge
    faster than society gathers wisdom.
    -- Isaac Asimov

    Roel Schroeven
     
    Roel Schroeven, Apr 6, 2008
    #17
  18. llothar

    Steve Holden Guest

    Re: When does a binary extension gets the file extension '.pyd' andwhen is it '.so'

    Fredrik Lundh wrote:
    > Steve Holden wrote:
    >
    >>> for reference, here's what I get on Ubuntu 7.10, with the standard
    >>> Python interpreter (2.5.1):
    >>>
    >>> $ python -c "import imp; print imp.get_suffixes()"
    >>> [('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
    >>> ('.pyc', 'rb', 2)]
    >>>
    >>> any Ubuntu gurus here that can sort this one out?
    >>>

    >> I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
    >> team decide that you would be able to import extension module YYY either
    >> from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
    >> to answer that I have no idea at all.

    >
    > oh, the ".so" and "module.so" is standard Python behaviour (see my first
    > post in this thread). what I cannot figure out is how "llothar" has
    > managed to get setup.py to build extensions that an Ubuntu Python cannot
    > load, without noticing.
    >

    Well, at least *I* learned something in this thread. I had missed that
    second paragraph in your first post, and hadn't realised it from other
    sources.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Apr 7, 2008
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Stefan Mueller
    Replies:
    5
    Views:
    551
    Steven Saunderson
    Jul 10, 2006
  2. Jon Nicoll
    Replies:
    0
    Views:
    289
    Jon Nicoll
    Aug 12, 2004
  3. Toby
    Replies:
    2
    Views:
    1,429
    Dennis Lee Bieber
    Oct 31, 2006
  4. Allen
    Replies:
    12
    Views:
    541
    Laurent Pointal
    Jun 14, 2007
  5. Dave Guthrie
    Replies:
    0
    Views:
    348
    Dave Guthrie
    May 11, 2010
Loading...

Share This Page