Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP

[Alf P. Steinbach /Usenet]

* Martin v. Loewis, on 07.07.2010 21:56:

I was guessing that at one time there was no PyMem_Malloc. And that it
was introduced to fix Windows-specific problems, but inadvertently
without updating the macro. It's just a guess as to reasons why the
macro uses malloc directly.

It might indeed be that the function version was introduced specifically
for Windows. However, the macro was left intentionally: both for
backwards compatibility, and for use inside Python itself.

Except for the problems with file descriptors I think a practical
interim solution for extensions implemented in C could be to just link
the runtime lib statically.

[...]

When I wrote "link the runtime lib statically" that was an alternative
to the usual link-as-DLL.

Ok, I lost the thread. When you said "a practical interim solution"
you were talking about what problem? I thought the discussion was
about the need to link with the same DLL version as Python.

The main problem that the required MSVC redistributables are not necessarily
present on the end user's system.

However, it would surely make sense to link with a different DLL than
the one that Python links with, assuming that would actually work.


It's actually straight-forward (or used to be, until they came up with
the SxS madness). It was actually the case that people did so
unexpectingly, and it seemed to work fine, except that it crashed when
passing FILE*. Then we started explaining that mixing CRTs is risky.

Oh.

Well then.


Cheers,

- Alf

 

[sturlamolden]

However, the C standard is silent wrt. to PyMem_MALLOC, and it certainly
allows the definition of macros which use the macro arguments more than
once.

Ok, I knew there was something odd here. PyMem_Malloc is indeed a
function, whilst PyMem_MALLOC is a deprecated macro.

 

[Grant Edwards]

Also observe that this macro is very badly written (even illegal) C.
Consider what this would do:

PyMem_MALLOC(n++)

According to Linus Thorvalds using macros like this is not even legal
C:

http://www.linuxfocus.org/common/src/January2004_linus.html

[Please don't use "legal" wrt. programs - it's not "illegal" to violate
the language's rules; you don't go to jail when doing so. Linus said
"not allowed"]

Nonsense.

The world "illegal" doesn't mean "you'll go to jail".

"Legal" and "illegal" are used to indicate conformance or
nonconformace with respect to some set of rules -- be they a
programming language standard, FIFA footbal rules, or Forumula 1
technical regulations.

It's perfectly standard usage to refer to an "illegal forward pass" in
American football, to "illegal tires" used during a race, or to an
"illegal operation" in a program.

 

[Alf P. Steinbach /Usenet]

* Christian Heimes, on 07.07.2010 22:47:

It's not a problem for Python anymore. It took a while to sort all
problems out. Martin and other developers have successfully figured out
how to install the CRT for system wide and local user installations. A
system wide installation installs the CRT in the side by side cache
(WinSxS). A local installation keeps the msvcrt90.dll and the
Microsoft.VC90.CRT.manifest next to the python.exe. Python extensions no
longer embed a manifest so they share the CRT from the python.exe process.

In order to ship a standalone exe you have to keep the CRT next to your
exe. This should work for py2exe binaries as well. At our company we
install our application stack entirely from subversion including Python
2.6.5, Sun JRE and lots of other stuff. This works perfectly fine for us
even for servers without the MSVCRT redistributable.

I think you're talking about a different problem. The CRT installed along with
CPython works for extensions using the MSVC 9.0 CRT.

However developing an extension with MSVC 10 the extension will use the 10.0
CRT, which is not necessarily present on the end user's system.

As I see it there are five solutions with different trade-offs:

A Already having Visual Studio 2008 (MSVC 9.0), or coughing up the
money for an MSDN subscription, or visiting trade shows, so as to
obtain that compiler version.
-> Not an option for everybody.

B Linking the CRT statically.
-> Increased size, problems with CRT state such as file descriptors.

C Linking the CRT dynamically and bundling the MSVC redistributables
with the extension.
-> Even more increased size for the download, but smaller total
footprint for extensions in sum; same CRT state problems.

D Linking the CRT dynamically and providing an optional download and
install of the redistributables if they're not present. This would
best be done with some support from the Python installation machinery.
-> Small nice size for extensions, still same CRT state problems.

E As D + a new compiler-independent native code interface that
does not carry dependencies on CRT state such as file descriptors, like JNI.
-> Really huge effort, and cannot be applied until some new Python version.

And I think the clue here is that the CRT state problems can be avoided by
careful coding.

Hence, for those who cannot do A I think B is a realistic practical option, and
D would be nice...


Cheers,

- Alf

 

[Alf P. Steinbach /Usenet]

* Alf P. Steinbach /Usenet, on 07.07.2010 23:19:

However developing an extension with MSVC 10 the extension will use the
10.0 CRT, which is not necessarily present on the end user's system.

As I see it there are five solutions with different trade-offs:

A Already having Visual Studio 2008 (MSVC 9.0), or coughing up the
money for an MSDN subscription, or visiting trade shows, so as to
obtain that compiler version.
-> Not an option for everybody.

B Linking the CRT statically.
-> Increased size, problems with CRT state such as file descriptors.

C Linking the CRT dynamically and bundling the MSVC redistributables
with the extension.
-> Even more increased size for the download, but smaller total
footprint for extensions in sum; same CRT state problems.

D Linking the CRT dynamically and providing an optional download and
install of the redistributables if they're not present. This would
best be done with some support from the Python installation machinery.
-> Small nice size for extensions, still same CRT state problems.

E As D + a new compiler-independent native code interface that
does not carry dependencies on CRT state such as file descriptors, like
JNI.
-> Really huge effort, and cannot be applied until some new Python version.

And I think the clue here is that the CRT state problems can be avoided
by careful coding.

Hence, for those who cannot do A I think B is a realistic practical
option, and D would be nice...

Wait...

F Possibly, as the docs say,

"Developer Studio will throw in a lot of import libraries that you do not really
need, adding about 100K to your executable. To get rid of them, use the Project
Settings dialog, Link tab, to specify ignore default libraries. Add the correct
msvcrtxx.lib to the list of libraries."

Can anyone confirm whether this works in practice with MSVC 10?


Cheers,

- Alf

 

[sturlamolden]

   D Linking the CRT dynamically and providing an optional download and
     install of the redistributables if they're not present. This would
     best be done with some support from the Python installation machinery.
     -> Small nice size for extensions, still same CRT state problems.

This was a problem for py2exe'd apps before Python 2.6 (i.e. no public
download for Visual C++ 2003 runtime files.) But for Visual C++ 2008
and 2010, the CRTs can be downloaded from Microsoft and need not be
shipped with the application.

http://www.microsoft.com/downloads/...34-3E03-4391-8A4D-074B9F2BC1BF&displaylang=en
http://www.microsoft.com/downloads/...CA-337F-4B40-8C14-157CFDFFEE4E&displaylang=en

http://www.microsoft.com/downloads/...5e-6de6-4d3a-a423-37bf0912db84&displaylang=en
http://www.microsoft.com/downloads/...9E-43C8-4655-81BF-9350143D5867&displaylang=en

 

[Jonathan Hartley]

sturlamolden has already given the primary reason: Python,
traditionally, attempts to use and work with the system vendor's
compiler. On Windows, that's MSC. It's typically the one that best knows
about platform details that other compilers might be unaware of.

In addition, it's also the compiler and IDE that Windows developers (not
just Python core people, but also extension developers and embedders)
prefer to use, as it has quite good IDE support (in particular debugging
and code browsing).

Perhaps more importantly, none of the other compilers is really an
alternative. GCC in particular cannot build the Win32 extensions, since
it doesn't support the COM and ATL C++ features that they rely on (and
may not support other MSC extensions, either). So the Win32 extensions
must be built with VS, which means Python itself needs to use the same
compiler.

Likewise important: gcc/mingw is *not* a complete C compiler on Windows.
A complete C compiler would have to include a CRT (on Windows); mingw
doesn't (cygwin does, but I think you weren't proposing that Python be
built for cygwin - you can easily get cygwin Python anyway). Instead,
mingw relies on users having a CRT available to
them - and this will be a Microsoft one. So even if gcc was used, we
would have versioning issues with Microsoft CRTs, plus we would have to
rely on target systems including the right CRT, as we couldn't include
it in the distribution.

HTH,
Martin

I see. Thanks very much to both of you for the info, much appreciated.

 

[David Cournapeau]

My preferred long-term solution is to reduce the usage of the C library
in CPython as much as reasonable, atleast on Windows. Memory management
could directly use the heap functions (or even more directly
VirtualAlloc); filenos could be OS handles, and so on. There are
probably limitations to what you can achieve, but I think it's worth trying.

I saw you already mentioned work toward this a few months (years ?)
ago. Is there some kind of roadmap, or could you use some help ? I
would really like to solve this issue as much as we possibly can,

David

 

[Martin v. Löwis]

I saw you already mentioned work toward this a few months (years ?)

ago. Is there some kind of roadmap, or could you use some help ? I
would really like to solve this issue as much as we possibly can,

Well, Python 3 has already dropped stdio for its own io library, and
I do want to get rid of the remaining FILE* usage for 3.2. Any other
change needs to be discussed on python-dev first; contributions are welcome.

Regards,
Martin

 

[Stephen Hansen]

Well, Python 3 has already dropped stdio for its own io library, and
I do want to get rid of the remaining FILE* usage for 3.2. Any other
change needs to be discussed on python-dev first; contributions are welcome.

Really? I wasn't entirely aware of this effect of the "io" module.
Somehow, without at all paying attention (because certain third party
modules blocking me for awhile, I never looked close enough), I just
sort of thought the "io" module was mostly thin wrappers around stdio
primitives, into a more Pythonic API.

This actually makes me long for Python 3 more. And adjust agendas to
push my day-job work to 2.7 as soon as possible, and start a preemptive
reconfiguration to support a future Py3k migration.

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJMNWJeAAoJEKcbwptVWx/l3moH/2qlJpDTddhe/4rqsT8HBP7N
J5+3jk41ekZfs9HWixHAhZT5WlrM88XBCpO8yiXLa9zECKCIMYypOKI/aI+6lsyF
V2+u6VTpFD167tvK7YD4QRZ7tE/CfagkB9wSJzazaqIY4ChwOgoUKtBuPwHOJBvn
M/gy68/kfr03wOUGyDKNhzetVrqMAqyt1I4/wKfrtWn8D9+TW9jXgyXp1lGqk7ov
beZzt9il9tznrcBVXmTd4FiowThz64vTSBHjqtfh6ujIsb3m82mzg+BHsWgbultb
Jz6DzcgQmh6qWJPdMkV6pyAdPdILYwE+MApWnoxPXzRLkNSAkKgywZWLb/8JQVI=
=sMMR
-----END PGP SIGNATURE-----

 

[Neil Hodgson]

sturlamolden:

Windows did this too (msvcrt.dll) up to the VS2003 release, which came
with msvcr71.dll in addition. Since then, M$ (pronounced Megadollar
Corp.) have published msvcr80.dll, msvcr90.dll, and msvcr100.dll (and
corresponding C++ versions) to annoy C and C++ developers into
converting to C# .NET. (And yes, programs using third-party DLL and
OCX components become unstable from this. You have to check each DLL/
OCX you use, and each DLL used by each DLL, etc. How fun...)

One of the benefits to COM is that it acts as a C runtime firewall -
it has its own memory allocation interface (IMalloc / CoGetMalloc) and
file I/O is performed through interfaces like IStream. It is quite
common to use an OCX compiled with one compiler in an application
compiled with another.

If you break the rules by using malloc rather than IMalloc for memory
that is deallocated by a different component to that which allocated it
or try to pass around FILE* objects then you will see failures. So,
always follow the COM rules.

Neil

 

[sturlamolden]

   If you break the rules by using malloc rather than IMalloc for memory
that is deallocated by a different component to that which allocated it
or try to pass around FILE* objects then you will see failures.

Yes, the CRT issue applies to COM as well. COM developers are even
less aware of this than Python developers.

So,
always follow the COM rules.

Or just avoid COM...

 

[Lawrence D'Oliveiro]

There are also utilities for mounting ISOs directly without burning them
to a physical disk.

You need special utilities to do this??

 

[Robert Kern]

You need special utilities to do this??

On at least some versions of Windows, Yes.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Grant Edwards]

You need special utilities to do this??

Not if the OS and VFS are competently designed. In Linux all you need
to do is this:

mount -o loop /path/to/file.iso /mount/point

Apparently you've got to jump through all sorts of hoops using 3rd
party software to do something analgous in MS Windows.

 

[Zooko O'Whielacronx]

I presume this problem would go away if future versions of Python
itself were compiled on Windows with something like MinGW gcc.

You might want to track issue3871. Roumen Petrov has done a lot of
work to make CPython compilable with mingw on Windows, as well as to
make it possible to compile CPython on a different operating system
and produce a CPython executable for Windows (cross-compile).

And by the way, I've usually had success building my native extension
modules with mingw. I understand (vaguely) that if a native extension
module needs to pass FILE*'s or C++ iostreams back and forth to
different extension modules or the the core CPython interpreter then
this could lead to segfaults, but none of my extension modules need to
do that.

I would suggest that people try to build their native extension
modules with mingw, and if it doesn't work report a bug (to mingw
project and to the Python project) so that we can track more precisely
what the issues are.

Regards,

Zooko

http://bugs.python.org/issue3871# cross and native build of python for
mingw32 with distutils

 

[David Cournapeau]

I would suggest that people try to build their native extension
modules with mingw, and if it doesn't work report a bug (to mingw
project and to the Python project) so that we can track more precisely
what the issues are.

To be clear, building extensions with mingw for the official python
works well. Numpy and scipy official binaries have been built with
mingw for years. There are problems for 64 bits binaries, though

David

 

[Burton Samograd]

Not if the OS and VFS are competently designed. In Linux all you need
to do is this:

mount -o loop /path/to/file.iso /mount/point

Apparently you've got to jump through all sorts of hoops using 3rd
party software to do something analgous in MS Windows.

In Windows you use DaemonTools.

 

[Robin Becker]

In Windows you use DaemonTools.

I use VCdControlTool.exe which is some kind of M$ utility, but perhaps that
doesn't work for everyone.

 

[Jean-Michel Pichavant]

On at least some versions of Windows, Yes.

You need the "mount" utility with Linux.

JM