When does a binary extension gets the file extension '.pyd' and whenis it '.so'

L

llothar

On windows everything is '.pyd' but there seems to be two ways to get
this on unix?
Why and what is the rule?
 
F

Fredrik Lundh

llothar said:
On windows everything is '.pyd' but there seems to be two ways to get
this on unix?

If you attempt to import the module "spam" on Windows, Python looks for
"spam.dll" and "spam.pyd" (in addition to "spam.py/spam.pyw/spam.pyc" etc)

On most Unix platforms, Python looks for "spam.so" and "spammodule.so".

You can check what suffixes a given Python version uses via the "imp"
module:

To see *where* Python is looking as well, use the "-vv" flag:

$ python -vv -c "import spam"
...
# trying spam.so
# trying spammodule.so
# trying spam.py
# trying spam.pyc
... etc

(-vv prints loads of stuff, so you may want to use "grep" to filter out
the stuff you're interested in:

$ python -vv -c "import spam" 2>&1 | grep spam

or, under Windows:
> python -vv -c "import spam" 2> out.txt
> findstr spam out.txt
)

Why and what is the rule?

If you want Python to be able to import your binary extension, make sure
to use a name it looks for.

</F>
 
L

llothar

Thanks, my question was not how can i make python to it find. I don't
have a problem.

My question was: Why does setup.py generated sometimes a pyd and
sometimes a so file?

There must be a rule behind this.

Unforunately setup.py is not well documented. Here i mean i need a
specification not a
tutorial, because i want to know something not do something.

Don't ask why i need to know: I need to know. And maybe somebody here
can drop a
line before i have to dig around in the source code.
 
F

Fredrik Lundh

llothar said:
My question was: Why does setup.py generated sometimes a pyd and
sometimes a so file?

setup.py picks an extension that happens to work on the platform you're
running setup.py on. doing otherwise would be pretty pointless.

</F>
 
L

llothar

setup.py picks an extension that happens to work on the platform you're
running setup.py on. doing otherwise would be pretty pointless.

</F>

Unfortunately as pointless as the answers i got so far.


Okay i try it one more time:

I ship an application that compiles an python interpreter and
extension on a remote system.
It also needs to copy this created items around. So if i use setup.py
to create an
extension i need to know the file name of the generated file.

Damned this is trivial and a fundamental question and it is not
documented anywhere.

I have a clue at the moment that it might be ".so" when python is
compiled without shared library
and ".pyd" otherwise (configure option --enable-shared) . But this is
just a guess. Does anybody know?

And by the way: I think this is a bug and should be fixed. If the
platform does allow renaming the
extension of a DLL (does HP/UX allow this?) it should always be
".pyd"
 
F

Fredrik Lundh

llothar said:
> I ship an application that compiles an python interpreter and
> extension on a remote system.
> It also needs to copy this created items around. So if i use setup.py
> to create an
> extension i need to know the file name of the generated file.

so why not just ask setup.py to copy the files for you?

you can either use "install" with the --home option to copy all the
files to a given directory, or create a binary kit with "bdist" and
unpack the resulting archive at the target location.
> Unfortunately as pointless as the answers i got so far.

well, cutting the "I'm not going to tell you why I need to know this"
stuff might help people come up with solutions to your actual problem
instead of posting stuff that's informative but misses the point.

</F>
 
S

Steve Holden

llothar said:
Unfortunately as pointless as the answers i got so far.
Right, so you think people aren't trying to help you?

Put your paranoia back in your pocket :)
Okay i try it one more time:

I ship an application that compiles an python interpreter and
extension on a remote system.
It also needs to copy this created items around. So if i use setup.py
to create an
extension i need to know the file name of the generated file.

Damned this is trivial and a fundamental question and it is not
documented anywhere.

I have a clue at the moment that it might be ".so" when python is
compiled without shared library
and ".pyd" otherwise (configure option --enable-shared) . But this is
just a guess. Does anybody know?

And by the way: I think this is a bug and should be fixed. If the
platform does allow renaming the
extension of a DLL (does HP/UX allow this?) it should always be
".pyd"

You display your ignorance here. The ".pyd" extension is used on Windows
as an alternative to ".dll", but both are recognized as shared
libraries. Personally I'm not really sure why they even chose to use
".pyd", which is confusing to most Windows users. In UNIX/Linux
environments ".so" is the standard extension for a shared library.

To depart from the platform standard would be unhelpful and confusing to
the majority of users. It's know use telling us what you think: tell us
instead the compelling reasons why your opinion is correct. Opinions,
after all, are so cheap that everyone can have one.

There are ways to build distributions of Python extensions (modules or
packages involving binary code from languages like C or C++), but you
will want to understand a bit more about computing in general (and work
on your social skills ;-) before you start to approach them.

regards
Steve
 
F

Fredrik Lundh

Steve said:
You display your ignorance here. The ".pyd" extension is used on Windows
as an alternative to ".dll", but both are recognized as shared
libraries. Personally I'm not really sure why they even chose to use
".pyd", which is confusing to most Windows users. In UNIX/Linux
environments ".so" is the standard extension for a shared library.

background:

http://mail.python.org/pipermail/python-dev/1999-December/001456.html

"Python cannot import every .dll file. Only .dll files that conform to
the convention for Python extension modules can be imported. (The
convention is that it must export an init<module> function.)

On most other platforms, shared libraries must have a specific
extension (e.g. .so on most Unix). Python allows you to drop such a
file into any directory where is looks for modules, and it will then
direct the dynamic load support to load that specific file.

This seems logical -- Python extensions must live in directories that
Python searches (Python must do its own search because the search
order is significant).

On Windows, Python uses the same strategy. The only modification is
that it is allowed to give the file a different extension, namely
..pyd, to indicate that this really is a Python extension and not a
regular DLL. This was mostly introduced because it is apparently
common to have an existing DLL "foo.dll" and write a Python wrapper
for it that is also called "foo". Clearly, two files foo.dll are too
confusing, so we let you name the wrapper foo.pyd. But because the
file format is essentially that of a DLL, we don't *require* this
renaming; some ways of creating DLLs in the first place may make it
difficult to do."

</F>
 
L

llothar

Right, so you think people aren't trying to help you?

I think they are not reading the question.
You display your ignorance here. The ".pyd" extension is used on Windows
as an alternative to ".dll", but both are recognized as shared
libraries. Personally I'm not really sure why they even chose to use
".pyd", which is confusing to most Windows users.

Here i agree. But having it's own identifiying extension has also some
small benefits.
To depart from the platform standard would be unhelpful and confusing to
the majority of users. It's know use telling us what you think: tell us
instead the compelling reasons why your opinion is correct. Opinions,
after all, are so cheap that everyone can have one.

Because i want a uniform solution. Either use "dllso" or use "pyd"
but
stay with one decision once made. At the moment when i build python on
my ubuntu system without "--enable-shared" i get a pyd file created,
if
i use "--enable-shared" it is a so file.

I don't know if this is a special case on Linux or the general on unix
systems
(i only have Linux, FreeBSD, NetBSD, OpenBSD, Solaris and MacOSX in
mind).
There are ways to build distributions of Python extensions (modules or
packages involving binary code from languages like C or C++), but you
will want to understand a bit more about computing in general

Believe me nobody needs to teach me anything about general programming
anymore.
(and work on your social skills ;-)

I don't think so. I asked a pretty simple question and as usual on
usenet nobody
read the question but answered to complete different topics. Answers
on usenet are
so cheap, everybody likes to give one - no matter if it is ontopic,
right or wrong.
And this does not really help.

My question is simple and person who knows setup.py and distools would
be able to
give the answer in a small sentence if there is a strategy behind it
and it's not
only a bug.

Unfortunately there is no python.core mailing list that i know so i
ask here.
 
F

Fredrik Lundh

llothar said:
I don't think so. I asked a pretty simple question and as usual on
usenet nobody read the question

did *you* read your own question? it took you three posts before you
mentioned what you were trying to do, and four posts before you bothered
to mention that you're seeing this behaviour on Ubuntu.

and for the record, Python doesn't look for PYD files on any of the Unix
boxes I have convenient access to right now. what Ubuntu version are
you using, what Python version do you have, and what does

$ python -c "import imp; print imp.get_suffixes()"

print on your machine?

</F>
 
S

Steve Holden

llothar wrote:
[...]
Unfortunately there is no python.core mailing list that i know so i
ask here.
Well your researches can't have been that extensive: the developers live
on python-dev@pythonlorg, otherwise known as comp.land.python-dev. But
you will need to ask your question rather more carefully there to het a
meaningful answer. Good luck.

regards
Steve
 
F

Fredrik Lundh

Fredrik said:
and for the record, Python doesn't look for PYD files on any of the Unix
boxes I have convenient access to right now. what Ubuntu version are
you using, what Python version do you have, and what does

$ python -c "import imp; print imp.get_suffixes()"

print on your machine?

for reference, here's what I get on Ubuntu 7.10, with the standard
Python interpreter (2.5.1):

$ python -c "import imp; print imp.get_suffixes()"
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
('.pyc', 'rb', 2)]

any Ubuntu gurus here that can sort this one out?

</F>
 
M

Martin v. Löwis

There must be a rule behind this.

There are multiple rules behind this. Python normally uses
the same extension for shared libraries as the operating
system, as the operating system may refuse to load them if
it doesn't. So it *can't* use .pyd on Unix, because that
might not work (on some implementations of Unix).

It can't use .dll on Windows, because that may (and did)
conflict with existing DLLs, hence it uses .pyd.

Depending on the system, an extension module may have one
of these suffixes:
- .so
- module.so
- .sl
- module.sl
- .exe
- .EXE
- module.exe
- module.EXE
- .pyd
- _d.pyd

Regards,
Martin
 
S

Steve Holden

Fredrik said:
Fredrik said:
and for the record, Python doesn't look for PYD files on any of the Unix
boxes I have convenient access to right now. what Ubuntu version are
you using, what Python version do you have, and what does

$ python -c "import imp; print imp.get_suffixes()"

print on your machine?

for reference, here's what I get on Ubuntu 7.10, with the standard
Python interpreter (2.5.1):

$ python -c "import imp; print imp.get_suffixes()"
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
('.pyc', 'rb', 2)]

any Ubuntu gurus here that can sort this one out?
I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
team decide that you would be able to import extension module YYY either
from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
to answer that I have no idea at all.

regards
Steve
 
S

Steve Holden

Fredrik said:
Fredrik said:
and for the record, Python doesn't look for PYD files on any of the Unix
boxes I have convenient access to right now. what Ubuntu version are
you using, what Python version do you have, and what does

$ python -c "import imp; print imp.get_suffixes()"

print on your machine?

for reference, here's what I get on Ubuntu 7.10, with the standard
Python interpreter (2.5.1):

$ python -c "import imp; print imp.get_suffixes()"
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
('.pyc', 'rb', 2)]

any Ubuntu gurus here that can sort this one out?
I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
team decide that you would be able to import extension module YYY either
from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
to answer that I have no idea at all.

regards
Steve
 
F

Fredrik Lundh

Steve said:
for reference, here's what I get on Ubuntu 7.10, with the standard
Python interpreter (2.5.1):

$ python -c "import imp; print imp.get_suffixes()"
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
('.pyc', 'rb', 2)]

any Ubuntu gurus here that can sort this one out?
I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
team decide that you would be able to import extension module YYY either
from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
to answer that I have no idea at all.

oh, the ".so" and "module.so" is standard Python behaviour (see my first
post in this thread). what I cannot figure out is how "llothar" has
managed to get setup.py to build extensions that an Ubuntu Python cannot
load, without noticing.

</F>
 
R

Roel Schroeven

llothar schreef:
Believe me nobody needs to teach me anything about general programming
anymore.

"The people who are best at programming are the people who realize how
small their brains are. They are humble." -- Edsger Dijkstra, 1972

(http://www.codinghorror.com/blog/archives/000051.html,
http://www.cs.utexas.edu/users/EWD/ewd03xx/EWD340.PDF)


--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov

Roel Schroeven
 
S

Steve Holden

Fredrik said:
Steve said:
for reference, here's what I get on Ubuntu 7.10, with the standard
Python interpreter (2.5.1):

$ python -c "import imp; print imp.get_suffixes()"
[('.so', 'rb', 3), ('module.so', 'rb', 3), ('.py', 'U', 1),
('.pyc', 'rb', 2)]

any Ubuntu gurus here that can sort this one out?
I wouldn't claim to be an Ubuntu guru but it seems as though the Ubuntu
team decide that you would be able to import extension module YYY either
from YYY.so or from YYYmodule.so. IF you are asking *why* then I'd have
to answer that I have no idea at all.

oh, the ".so" and "module.so" is standard Python behaviour (see my first
post in this thread). what I cannot figure out is how "llothar" has
managed to get setup.py to build extensions that an Ubuntu Python cannot
load, without noticing.
Well, at least *I* learned something in this thread. I had missed that
second paragraph in your first post, and hadn't realised it from other
sources.

regards
Steve
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,906
Latest member
SkinfixSkintag

Latest Threads

Top