Problem with msvcrt60 vs. msvcr71 vs. strdup/free

Discussion in 'Python' started by Gerhard Haering, Dec 21, 2004.

  1. Hello,

    I used to build Python extension modules with mingw. Now, Python has
    switched to the MSVCR71 runtime with version 2.4, and I thought mingw
    has support for this. But I get problems with symbols being referenced
    from the wrong DLLs.

    You can see the problem by compiling this:

    ##################
    #include <string.h>

    int main()
    {
    char* s;
    int i;


    for (i = 0; i < 10; i++) {
    s = strdup("foo");
    free(s);
    }

    return 0;
    }
    ##################

    with gcc x.c -lmsvcr71

    Then if you run a.exe it crashes.

    If you use depends.exe on it, you see that it resolves strdup() via
    msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
    the one malloc, but free() a different free() from the other DLL,
    which is undoubtedly the reason for the crash.

    Is there any way I can force mingw to not link in msvcr for things
    like strdup?

    I think I found a very strange and ugly workaround: if I use _strdup()
    instead of strdup(), it is resolved using the msvcr71 DLL.

    I hope it is (soon) possible to use only the msvcr71 runtime from the
    mingw compiler.

    -- Gerhard
    --
    A: Because it messes up the order in which people normally read text.
    Q: Why is top-posting such a bad thing?
    A: Top-posting.
    Q: What is the most annoying thing on usenet and in e-mail?

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.4 (GNU/Linux)

    iD8DBQFByCmWdIO4ozGCH14RAu0nAJwKYhMLulwl52UltcFk00sy55yEpQCfaTsS
    Ba8Q5W8g23lRe/VM/K/kBCU=
    =L8+w
    -----END PGP SIGNATURE-----
     
    Gerhard Haering, Dec 21, 2004
    #1
    1. Advertising

  2. Gerhard Haering <> writes:

    > Hello,
    >
    > I used to build Python extension modules with mingw. Now, Python has
    > switched to the MSVCR71 runtime with version 2.4, and I thought mingw
    > has support for this. But I get problems with symbols being referenced
    > from the wrong DLLs.
    >
    > You can see the problem by compiling this:
    >
    > ##################
    > #include <string.h>
    >
    > int main()
    > {
    > char* s;
    > int i;
    >
    >
    > for (i = 0; i < 10; i++) {
    > s = strdup("foo");
    > free(s);
    > }
    >
    > return 0;
    > }
    > ##################
    >
    > with gcc x.c -lmsvcr71
    >
    > Then if you run a.exe it crashes.
    >
    > If you use depends.exe on it, you see that it resolves strdup() via
    > msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
    > the one malloc, but free() a different free() from the other DLL,
    > which is undoubtedly the reason for the crash.
    >
    > Is there any way I can force mingw to not link in msvcr for things
    > like strdup?


    Only guesswork, but replacing this section in the
    lib\gcc-lib\mingw32\3.2.3\specs file (I can only guess wht this file does)

    *libgcc:
    %{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcrt

    with this one:

    *libgcc:
    %{mthreads:-lmingwthrd} -lmingw32 -lgcc -lmoldname -lmingwex -lmsvcr71

    seemed to do the trick.

    Thomas
     
    Thomas Heller, Dec 21, 2004
    #2
    1. Advertising

  3. Gerhard Haering

    Guest

    From: "" <>
    Newsgroups: comp.lang.python
    Subject: Re: Problem with msvcrt60 vs. msvcr71 vs. strdup/free
    Date: Wed, 22 Dec 2004 11:07:02 -0800

    Gerhard Haering wrote:
    > Hello,
    >
    > I used to build Python extension modules with mingw. Now, Python has
    > switched to the MSVCR71 runtime with version 2.4, and I thought mingw
    > has support for this. But I get problems with symbols being

    referenced
    > from the wrong DLLs.
    >
    > You can see the problem by compiling this:
    >
    > ##################
    > #include <string.h>
    >
    > int main()
    > {
    > char* s;
    > int i;
    >
    >
    > for (i = 0; i < 10; i++) {
    > s = strdup("foo");
    > free(s);
    > }
    >
    > return 0;
    > }
    > ##################
    >
    > with gcc x.c -lmsvcr71
    >
    > Then if you run a.exe it crashes.
    >
    > If you use depends.exe on it, you see that it resolves strdup() via
    > msvcrt, but the rest with msvcr71.dll. That's why strdup() is using
    > the one malloc, but free() a different free() from the other DLL,
    > which is undoubtedly the reason for the crash.
    >
    > Is there any way I can force mingw to not link in msvcr for things
    > like strdup?



    Hello


    Believe it or not I do not get a crash here.

    Using objdump on a.exe I get:

    [CUT]

    00005014 00005084 00000000 00000000 000052b8 00005100

    DLL Name: msvcrt.dll
    vma: Hint/Ord Member-Name Bound-To
    51b4 81 _strdup

    00005028 00005090 00000000 00000000 000052f8 0000510c

    DLL Name: msvcr71.dll
    vma: Hint/Ord Member-Name Bound-To
    51c0 59 __getmainargs
    51d0 82 __p__environ
    51e0 105 __set_app_type
    51f4 154 _cexit
    5200 211 _fileno
    520c 222 _fmode
    5218 225 _fpreset
    5224 273 _iob
    522c 390 _onexit
    5238 435 _setmode
    5244 599 atexit
    5250 634 free
    5258 715 signal

    0000503c 000050cc 00000000 00000000 00005314 00005148

    DLL Name: msvcrt.dll
    vma: Hint/Ord Member-Name Bound-To
    5264 510 abort
    526c 537 fflush
    5278 546 fprintf
    5284 603 malloc

    00005050 00000000 00000000 00000000 00000000 00000000


    [CUT]


    Having said that, let me add that I know where you are coming from. And
    even though your a.exe did not crash with me now, I know that other
    dlls and programs, in my case, did.

    And let it be known that unfortunately the measures suggested by the
    good people here do not always work. Changing the entry in the specs
    file works most of the time, but not on mmap.pyd (of the official
    Python source distribution) for example, where a call to some _lseek of
    some sort (I forgot the exact call) is still linked from msvcrt.dll not
    from msvcr71. And yes the test_mmap crashes if I compile it that way.

    Also changing the specs file seems to work only when the linking is
    simple. But if the linking gets complex (as in linking many objects
    files to build, say, pythonXY.dll) then even the specs change will not
    help. To illustrate, and in trying to get the pyMinGW built Python to
    link with msvcr71 that way, one ends up with a pythonXY.dll that
    references both runtime libraries: msvcrt, and msvcr71 with strange
    consequences. This particular version of Python crashes on site import
    hallucinating about (and I paraphrase!) "Null result from PY...XXX
    without Error."

    On MSDN, it is stated: "If your DLLs pass CRT resources across the
    msvcrt.dll and msvcr71.dll boundary, you will encounter issues with
    mismatched CRTs and need to recompile your project with Visual C++
    ..NET." [1]

    I have a gut feeling that "DLLs" here does not mean inter-dll passing
    of resources, but intra-dll passing of resources. In other words that
    if the self-same DLL uses both run-times then you are in trouble; but I
    don't think that this relates to the situation where one DLL is using
    msvcrt and another DLL is using msvcr71. Please bear with me.

    In this light, I truly don't know why would we try, or recommend
    others, to link with libmsvcr71.a, if some mix-up is still possible. I
    have yet to see a crash occur when Python24.dll or an extension of the
    same uses the old way of linking with libmsvcrt.a. That way we are
    assured that no references are made to msvcr71 when msvcrt is the one
    used when linking.

    I could be mistaken, but I think people are confusing the behavior of
    the MS compilers with MinGW's. The old MS compiler (V. 6) uses
    msvcrt.lib by default, and even when the newer MSVS uses the same name
    for the its default runtime, that msvcrt link implicitly means the
    newer msvcr71 [2]. This behind the scenes behavior is what seems to
    have people mixed up. You see if one has older projects compiled using
    V. 6, then linking these object files to other object files compiled
    with the newer MSVS could effectively mean that the resulting DLL will
    have references to both run-times, something by the way is also
    mentioned in the MSDN resources [3].

    But as far as I know this is not the case with regards to MinGW. MinGW
    explicitly uses msvcrt, and so old and newer projects should in theory
    have no problems if they stick to msvcrt, as opposed to going the MS
    compilers' path-- shooting ourselves in the foot-- and risking linking
    to two run-times at the same time.

    I have tried changing the specs, and changing the location of msvcr71
    and the results are never 100% re-assuring. And so I have resorted to
    linking with the good old msvcrt and I have not had a crash due to this
    to date. Of course this is my humble experience and I could be mistaken
    big time.

    For what it is worth, however, my extension DLLs are clean in that they
    reference msvcrt only and not msvcr71, and hence all the malloc, and
    free and what have you are never shared with any other runtime. My
    point being that lacking empirical evidence to the contrary, I think
    this is how making extensions to Python 2.4 with MinGW should be: using
    the default runtime libmsvcrt.a. It would be fantastic if the good
    people at MinGW, or others, can get that hard-wiring issue of the
    msvcrt resolved. But until that happens I think it is safer to extend
    Python 2.4 the old way. It seems to work fine.



    Regards,
    Khalid






    [1] C-RunTime Libraries from MSDN:
    http://tinyurl.com/q20a

    [2] Ibid.

    [3] Ibid.
     
    , Dec 22, 2004
    #3
  4. wrote:
    > Believe it or not I do not get a crash here.


    I can believe this. strdup does malloc, i.e. gets some
    memory from the heap, and free puts it back into the heap.
    Since this is a different CRT, it puts it back into
    a different heap. This is a leak, but it should not cause
    a crash.

    > I have a gut feeling that "DLLs" here does not mean inter-dll passing
    > of resources, but intra-dll passing of resources. In other words that
    > if the self-same DLL uses both run-times then you are in trouble; but I
    > don't think that this relates to the situation where one DLL is using
    > msvcrt and another DLL is using msvcr71. Please bear with me.


    I think you are wrong. What DLL invokes the CRT function does not matter
    - that the CRT functions come from different DLLs matters. Each CRT
    is loaded exactly once into an address space, regardless of how many
    DLLs refer to it. However, different CRTs have different sets of global
    variables, and this is what hurts.

    E.g. malloc use a pointer to the free list in the heap, and so does
    free. Each CRT has its own heap, and each has its own pointer to the
    free list. Then, doing malloc in one CRT and free in the other does
    not cause crashes (perhaps unless you use the debug CRT, which tries
    to determine whether it owns the pointer being freed). The CRT will
    put the memory into its own free list, and allocate it again if its
    own malloc asks for it. However, if you repeatedly do malloc() in one
    CRT and free() in another, you get a memory leak, since malloc() will
    not find the memory that has been freed.

    > In this light, I truly don't know why would we try, or recommend
    > others, to link with libmsvcr71.a, if some mix-up is still possible. I
    > have yet to see a crash occur when Python24.dll or an extension of the
    > same uses the old way of linking with libmsvcrt.a. That way we are
    > assured that no references are made to msvcr71 when msvcrt is the one
    > used when linking.


    This is very easy to create. Just do fopen() in an extension linked
    with msvcrt.dll, then do PyRun_AnyFile. This will crash, because fread()
    on a file opened in a different CRT will always crash.

    > I could be mistaken, but I think people are confusing the behavior of
    > the MS compilers with MinGW's. The old MS compiler (V. 6) uses
    > msvcrt.lib by default, and even when the newer MSVS uses the same name
    > for the its default runtime, that msvcrt link implicitly means the
    > newer msvcr71 [2].


    You are mistaken. The name of the import library is, and always was,
    msvcrt.lib. However, different versions of msvcrt.lib refer to different
    DLLs: msvcrt40.dll, msvcr70.dll, and msvcr71.dll.

    > This behind the scenes behavior is what seems to
    > have people mixed up. You see if one has older projects compiled using
    > V. 6, then linking these object files to other object files compiled
    > with the newer MSVS could effectively mean that the resulting DLL will
    > have references to both run-times, something by the way is also
    > mentioned in the MSDN resources [3].


    I think you are misreading [3]: It explicitly says

    "If you have a .lib or .obj file that needs to link to msvcrt.lib, then
    you should *not* have to recompile it to work with the new msvcrt.lib in
    Visual C++ .NET." (emphasis mine)

    The only case where you can link a single DLL with different CRTs using
    MS tools is when you combine different versions of the CRTs, e.g. debug
    and non-debug, or DLL and non-DLL.

    > But as far as I know this is not the case with regards to MinGW. MinGW
    > explicitly uses msvcrt, and so old and newer projects should in theory
    > have no problems if they stick to msvcrt, as opposed to going the MS
    > compilers' path-- shooting ourselves in the foot-- and risking linking
    > to two run-times at the same time.


    Python extensions should never ever link with msvcrt.dll. As your
    reference says:

    'The msvcrt.dll is now a "known DLL," meaning that it is a system
    component owned and built by Windows. It is intended for future use only
    by system-level components. An application should use and redistribute
    msvcr71.dll'

    Now, a Python extension is released from the need to distribute
    msvcr71.dll, since Python already does that. It should still link
    with that DLL.

    > I have tried changing the specs, and changing the location of msvcr71
    > and the results are never 100% re-assuring. And so I have resorted to
    > linking with the good old msvcrt and I have not had a crash due to this
    > to date. Of course this is my humble experience and I could be mistaken
    > big time.


    You are.

    > For what it is worth, however, my extension DLLs are clean in that they
    > reference msvcrt only and not msvcr71, and hence all the malloc, and
    > free and what have you are never shared with any other runtime.


    How do you know? What if your extension returns a pointer malloc'ed by
    msvcrt.dll, and returns it to python24.dll, and then python24.dll
    frees the pointer, using free() from msvcr71.dll?

    > My
    > point being that lacking empirical evidence to the contrary, I think
    > this is how making extensions to Python 2.4 with MinGW should be: using
    > the default runtime libmsvcrt.a. It would be fantastic if the good
    > people at MinGW, or others, can get that hard-wiring issue of the
    > msvcrt resolved. But until that happens I think it is safer to extend
    > Python 2.4 the old way. It seems to work fine.


    Indeed, it *will* work most of the time, as you rarely pass resources
    across CRTs in a typical Python extension. However, it is very hard
    to tell from the source code of the extension if such resource passing
    could ever happen (e.g. what about setlocale(), atexit(), malloc(),
    ....), so if you need something better than "seems to work", you really
    need to make sure that you understand all the issues, and have a build
    process that resolves them all.

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Dec 22, 2004
    #4
  5. Gerhard Haering

    Guest

    Martin v. Löwis wrote:
    > Indeed, it *will* work most of the time, as you rarely pass resources
    > across CRTs in a typical Python extension. However, it is very hard
    > to tell from the source code of the extension if such resource

    passing
    > could ever happen (e.g. what about setlocale(), atexit(), malloc(),
    > ...), so if you need something better than "seems to work", you

    really
    > need to make sure that you understand all the issues, and have a

    build
    > process that resolves them all.
    >
    > Regards,
    > Martin





    You know if I had not tried the example you gave, I'd probably still be
    arguing: "but it works." Oh well. Not anymore.

    For those who are interested, here is a patch to demo.c of the
    \python\dist\src\Demo\embed\demo.c demonstrating what Martin has argued
    about above. With such evidence at hand I guess it is safe to say that
    the MinGW die-hards will have to improvise. :)

    But seriously, and since I don't have the official Python 2.4
    distribution yet, and since the pyMinGW compiled Python I use relies on
    msvcrt, as opposed to msvcr71, I needed to reverse the roles here to
    see the problem first hand: making the "extension" demo use msvcr71 to
    see how it reacts with the home-made python24.dll which uses msvcrt.
    And sure enough: crash.

    Having said that, this seems to me not to be an issue for those who
    have or can build Python and/or the extensions they need from sources.
    Indeed recompiling the same "demo" to use msvcrt (to use the same
    run-time library that my python24.dll uses) instead of msvcr71 yields
    the expected correct results without crashes.

    And so I suspect this approach (compiling sources yourself) can shield
    people from this ugly mess, if they can build Python 2.4 and the
    extensions and have them (core and extension) use one and the same
    run-time library. But indeed this might be an issue for those who [1]
    use MinGW as their first choice and [2] cannot or won't bother to build
    their Python and/or extensions from sources, and hence have to rely on
    binary distributions.


    Regards
    Khalid



    --- demo.c Sat Nov 17 08:30:20 2001
    +++ demo2.c Thu Dec 23 02:02:40 2004
    @@ -34,6 +34,20 @@
    /* Note that you can call any public function of the Python
    interpreter here, e.g. call_object(). */

    + FILE *in;
    + int sts;
    +
    + if ((in = fopen(argv[1], "rt")) == NULL) {
    + fprintf(stderr, "Cannot open program file.\n");
    + return 1;
    + }
    +
    + printf("Working on %s\n", argv[1]);
    + sts = PyRun_AnyFile(in, argv[1]) != 0;
    + printf("sts: %d\n", sts);
    + fclose(in);
    +
    +
    /* Some more application specific code */
    printf("\nGoodbye, cruel world\n");
     
    , Dec 23, 2004
    #5
  6. wrote:
    > And so I suspect this approach (compiling sources yourself) can shield
    > people from this ugly mess, if they can build Python 2.4 and the
    > extensions and have them (core and extension) use one and the same
    > run-time library. But indeed this might be an issue for those who [1]
    > use MinGW as their first choice and [2] cannot or won't bother to build
    > their Python and/or extensions from sources, and hence have to rely on
    > binary distributions.


    I encourage anyone who gets further into solving the "How do I use
    MinGW to build Python2.4 (and later) standard-distribution compatible
    modules (.pyd s)?" question to share any clues they have. The MS
    free compiler is useful to many, but not all of us.

    I expect that MinGW is the most frequent choice for making Python
    modules in the "free compiler outside of MS" universe. Let's make
    that one work well, and (once we've done that) we can pass the lessons
    on for other compiler is desired.

    --Scott David Daniels


    p.s. I am able (and willing) to use the MS free compiler, but I have
    sympathy for those who aren't. I'd like to see it possible to use
    _lots_ of different compilers, but we are at the beginning of the
    problem, not the end. If Intel's compiler is as much faster than
    MS's as it used to be, I may eventually wind up using theirs anyway.
     
    Scott David Daniels, Dec 23, 2004
    #6
  7. Scott David Daniels wrote:
    > I encourage anyone who gets further into solving the "How do I use
    > MinGW to build Python2.4 (and later) standard-distribution compatible
    > modules (.pyd s)?" question to share any clues they have. The MS
    > free compiler is useful to many, but not all of us.


    I think the simplest answer is "It works out of the box". Just do

    python setup.py build --compiler=mingw32

    If it fails, report what the failure is (the first failure should
    be lack of libpython24.a, for which you can find build instructions
    in Google).

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Dec 23, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hoop

    Missing MSVCR71.dll

    Hoop, Aug 8, 2006, in forum: Python
    Replies:
    2
    Views:
    791
  2. sturlamolden
    Replies:
    7
    Views:
    495
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Jan 21, 2007
  3. Konte

    msvcr71.dll

    Konte, Feb 28, 2007, in forum: Python
    Replies:
    2
    Views:
    470
    Fredrik Lundh
    Feb 28, 2007
  4. Konte

    msvcr71.dll

    Konte, Feb 28, 2007, in forum: Python
    Replies:
    0
    Views:
    354
    Konte
    Feb 28, 2007
  5. priya

    Problem in Strdup()

    priya, Aug 23, 2005, in forum: C Programming
    Replies:
    37
    Views:
    1,141
Loading...

Share This Page