Building things with setup.py

Discussion in 'Python' started by James Stroud, Sep 22, 2006.

  1. James Stroud

    James Stroud Guest

    Hello All,

    This is annoying. I am trying to build scipy right now but every .so
    file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    to the ld flags.

    Main Question: When building with "setup.py", where can I include this
    information so I don't have to link every .so file by hand?

    Also, why would this not be caught by the developers of scipy and numpy?
    What factors go into needing to specify libraries and to properly link
    ..so files manually? Certainly the enthought didn't forget to test the
    build process, so why would my machine be special in this regard?

    I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
    libraries in non-standard places. Is this non-standard part what's
    killing me?

    James

    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #1
    1. Advertising

  2. James Stroud

    James Stroud Guest

    James Stroud wrote:
    > Hello All,
    >
    > This is annoying. I am trying to build scipy right now but every .so
    > file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    > to the ld flags.
    >
    > Main Question: When building with "setup.py", where can I include this
    > information so I don't have to link every .so file by hand?
    >
    > Also, why would this not be caught by the developers of scipy and numpy?
    > What factors go into needing to specify libraries and to properly link
    > .so files manually? Certainly the enthought didn't forget to test the
    > build process, so why would my machine be special in this regard?
    >
    > I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
    > libraries in non-standard places. Is this non-standard part what's
    > killing me?
    >
    > James
    >


    Well I added those arguments to $LDFLAGS, but it seemed like a pretty
    miserable hack, especially the "-shared" part.

    I'm still curious how to make a 3rd party setup.py work correctly.

    James

    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #2
    1. Advertising

  3. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > Hello All,
    >
    > This is annoying. I am trying to build scipy right now but every .so
    > file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    > to the ld flags.
    >
    > Main Question: When building with "setup.py", where can I include this
    > information so I don't have to link every .so file by hand?


    If it's actually necessary,

    python setup.py build_ext -lmylib

    > Also, why would this not be caught by the developers of scipy and numpy?


    Because we've never seen it before. There's no reason that you should have to
    add those libraries manually. Can you post your build log (the one that fails)
    to scipy-dev?

    http://www.scipy.org/Mailing_Lists

    > What factors go into needing to specify libraries and to properly link
    > .so files manually? Certainly the enthought didn't forget to test the
    > build process, so why would my machine be special in this regard?
    >
    > I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
    > libraries in non-standard places. Is this non-standard part what's
    > killing me?


    Possibly. Won't know more until I see the build log.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #3
  4. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > Hello All,
    >
    > This is annoying. I am trying to build scipy right now but every .so
    > file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    > to the ld flags.


    > I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
    > libraries in non-standard places. Is this non-standard part what's
    > killing me?


    It's possible that distutils is not finding the information about how the python
    interpreter was built. Did you build your own python(1)? Is the information in
    $PREFIX/lib/python2.4/config/ accurate?

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #4
  5. James Stroud

    James Stroud Guest

    Robert Kern wrote:
    > James Stroud wrote:
    >
    >> Hello All,
    >>
    >> This is annoying. I am trying to build scipy right now but every .so
    >> file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl
    >> -shared" to the ld flags.

    >
    >
    >> I'm running Linux FC4 on a dual intel p4 (~3.2 GHz) with sundry
    >> libraries in non-standard places. Is this non-standard part what's
    >> killing me?

    >
    >
    > It's possible that distutils is not finding the information about how
    > the python interpreter was built. Did you build your own python(1)? Is
    > the information in $PREFIX/lib/python2.4/config/ accurate?
    >


    I did build my own python 2.5, yesterday, requiring me to rebuild all
    extensions. Everything I do is compiled by hand as joe-user. I'm in a
    situation where I can't do RPM (and I don't have root on my work machine
    (theoretically ;-)) so, to be a good joe-user, everything I add goes
    into the prefix:

    $HOME/Programs

    This is the listing from $HOME/Programs/lib/python2.5/config:

    euler 6% ls
    total 4092
    8 config.c 12 install-sh* 44 Makefile 8 python.o
    8 Setup.config
    8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
    8 Setup.local

    The build process, by the way, required my copying libpython2.5.a to
    $HOME/Programs/lib.

    The text files Setup.config and Setup.local do not seem to have terribly
    specific information in them. Which file in particular should I inspect?

    I did not capture output from the build and I could not find a file with
    the word "log" in it that appears to be a build log. The jist of the
    problem is that first it can't find symbols from libpython2.5, then from
    libthread, etc. Then, it complains about no "MAIN__" when linking the
    ..so files with g77 and no "main" with gcc (which is curious), so I must
    include the -shared flag, after including -llibrary type flags for all
    of the libraries it doesn't know about.

    If its necessary, I can run setup.py build again and send the output to
    a file and post that to scipy-dev if you think it might be helpful.

    James

    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #5
  6. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > I did build my own python 2.5, yesterday, requiring me to rebuild all
    > extensions.


    Do other extensions build correctly? If so, it's beginning to look like a
    problem in numpy.distutils .

    > Everything I do is compiled by hand as joe-user. I'm in a
    > situation where I can't do RPM (and I don't have root on my work machine
    > (theoretically ;-)) so, to be a good joe-user, everything I add goes
    > into the prefix:
    >
    > $HOME/Programs
    >
    > This is the listing from $HOME/Programs/lib/python2.5/config:
    >
    > euler 6% ls
    > total 4092
    > 8 config.c 12 install-sh* 44 Makefile 8 python.o
    > 8 Setup.config
    > 8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
    > 8 Setup.local
    >
    > The build process, by the way, required my copying libpython2.5.a to
    > $HOME/Programs/lib.


    Hmm. That doesn't quite sound right, but it's been a while since I compiled the
    interpreter from source.

    > The text files Setup.config and Setup.local do not seem to have terribly
    > specific information in them. Which file in particular should I inspect?


    Makefile has most of that information. You can verify that distutils is finding
    it like so:

    >>> from distutils import sysconfig
    >>> sysconfig.get_makefile_filename()

    '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile'
    >>> d = sysconfig.parse_makefile(_)
    >>> import pprint
    >>> pprint.pprint(d)

    {'AR': 'ar',
    ...
    }

    > I did not capture output from the build and I could not find a file with
    > the word "log" in it that appears to be a build log. The jist of the
    > problem is that first it can't find symbols from libpython2.5, then from
    > libthread, etc. Then, it complains about no "MAIN__" when linking the
    > .so files with g77 and no "main" with gcc (which is curious), so I must
    > include the -shared flag, after including -llibrary type flags for all
    > of the libraries it doesn't know about.


    What versions of gcc and g77 are you using?

    > If its necessary, I can run setup.py build again and send the output to
    > a file and post that to scipy-dev if you think it might be helpful.


    Yes, that is what I intended.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #6
  7. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > Well I added those arguments to $LDFLAGS, but it seemed like a pretty
    > miserable hack, especially the "-shared" part.


    Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing) before?
    I.e. did you have something like this?

    export LDFLAGS=""
    python setup.py build

    If such environment variables are present, I think that numpy.distutils may
    clobber the values that are inferred from the configuration.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #7
  8. James Stroud schrieb:
    > This is annoying. I am trying to build scipy right now but every .so
    > file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    > to the ld flags.


    That shouldn't be necessary. Linking without this should work just fine.

    That way, since you made libpython2.5.a a static library, you link an
    entire Python interpreter into each and every extension module. This
    should not be done.

    What happens if you omit these flags?

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Sep 22, 2006
    #8
  9. James Stroud

    James Stroud Guest

    Robert Kern wrote:
    > James Stroud wrote:
    >> I did build my own python 2.5, yesterday, requiring me to rebuild all
    >> extensions.

    >
    > Do other extensions build correctly? If so, it's beginning to look like
    > a problem in numpy.distutils .


    It seems that every thing has built without incident (numarray, numeric,
    mxtexttools, pil, reportlab, just to name a few), with the exception,
    strangely enough, of python2.5 itself. Python required hiding previous
    python installations.

    >> Everything I do is compiled by hand as joe-user. I'm in a situation
    >> where I can't do RPM (and I don't have root on my work machine
    >> (theoretically ;-)) so, to be a good joe-user, everything I add goes
    >> into the prefix:
    >>
    >> $HOME/Programs
    >>
    >> This is the listing from $HOME/Programs/lib/python2.5/config:
    >>
    >> euler 6% ls
    >> total 4092
    >> 8 config.c 12 install-sh* 44 Makefile 8 python.o
    >> 8 Setup.config
    >> 8 config.c.in 3960 libpython2.5.a 12 makesetup* 24 Setup
    >> 8 Setup.local
    >>
    >> The build process, by the way, required my copying libpython2.5.a to
    >> $HOME/Programs/lib.

    >
    > Hmm. That doesn't quite sound right, but it's been a while since I
    > compiled the interpreter from source.


    Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
    was necessary to build numpy and scipy.

    >> The text files Setup.config and Setup.local do not seem to have
    >> terribly specific information in them. Which file in particular should
    >> I inspect?

    >
    > Makefile has most of that information. You can verify that distutils is
    > finding it like so:
    >
    > >>> from distutils import sysconfig
    > >>> sysconfig.get_makefile_filename()

    > '/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/config/Makefile'
    >
    > >>> d = sysconfig.parse_makefile(_)
    > >>> import pprint
    > >>> pprint.pprint(d)

    > {'AR': 'ar',
    > ...
    > }


    This output from the check is here:
    http://www.jamesstroud.com/build-scipy-logs/distutil-check.txt

    I'm not sure if this is relevant to what you see in the check, but I
    installed scipy_distutils AFTER installing scipy.

    >> I did not capture output from the build and I could not find a file
    >> with the word "log" in it that appears to be a build log. The jist of
    >> the problem is that first it can't find symbols from libpython2.5,
    >> then from libthread, etc. Then, it complains about no "MAIN__" when
    >> linking the .so files with g77 and no "main" with gcc (which is
    >> curious), so I must include the -shared flag, after including
    >> -llibrary type flags for all of the libraries it doesn't know about.

    >
    > What versions of gcc and g77 are you using?


    euler 2% g77 -v
    Reading specs from
    /auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
    Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
    (reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
    Thread model: posix
    gcc version 3.4.2
    euler 3% gcc -v
    Reading specs from
    /auto_nfs/data10/users/jstroud/Programs/bin/../lib/gcc/i686-pc-linux-gnu/3.4.2/specs
    Configured with: ./configure --prefix=/data1/users/jstroud/Programs :
    (reconfigured) ./configure --prefix=/data1/users/jstroud/Programs
    Thread model: posix
    gcc version 3.4.2

    >> If its necessary, I can run setup.py build again and send the output
    >> to a file and post that to scipy-dev if you think it might be helpful.

    >
    > Yes, that is what I intended.
    >


    This is posted here:
    http://www.jamesstroud.com/build-scipy-logs/build-scipy-py2.5-euler.txt

    For symmetry, similar output for numpy is posted here:
    http://www.jamesstroud.com/build-scipy-logs/build-numpy-py2.5-euler.txt

    Please let me know if I can be of any further help with these issues.

    James
     
    James Stroud, Sep 22, 2006
    #9
  10. James Stroud

    James Stroud Guest

    Martin v. Löwis wrote:
    > James Stroud schrieb:
    >> This is annoying. I am trying to build scipy right now but every .so
    >> file requires my adding "-lpython2.5 -lpthread -lm -lutil -ldl -shared"
    >> to the ld flags.

    >
    > That shouldn't be necessary. Linking without this should work just fine.


    Unfortunately, this was not my experience.

    > That way, since you made libpython2.5.a a static library, you link an
    > entire Python interpreter into each and every extension module. This
    > should not be done.


    Actually, a straight configure, make, install of the python2.5 source
    produced this library.

    > What happens if you omit these flags?


    Please see my last message to Robert Kern.

    James
     
    James Stroud, Sep 22, 2006
    #10
  11. James Stroud

    James Stroud Guest

    Robert Kern wrote:
    > James Stroud wrote:
    >> I did build my own python 2.5, yesterday, requiring me to rebuild all
    >> extensions.

    >
    > Do other extensions build correctly? If so, it's beginning to look like
    > a problem in numpy.distutils .


    Actually, I just found that MySQLdb had a similar problem with zlib, but
    instead of bonking during linking, it complained about the compress
    symbol when trying to import into python. The fix was adding "-lz" for
    the linking step, after which it imported just fine.

    James
     
    James Stroud, Sep 22, 2006
    #11
  12. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > Robert Kern wrote:
    >> James Stroud wrote:


    >>> The build process, by the way, required my copying libpython2.5.a to
    >>> $HOME/Programs/lib.

    >> Hmm. That doesn't quite sound right, but it's been a while since I
    >> compiled the interpreter from source.

    >
    > Sorry. To clarify, making libpython2.5a available in a $LD_LIBRARY_PATH
    > was necessary to build numpy and scipy.


    I don't see how that could have affected anything. Static libraries aren't
    looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

    > I'm not sure if this is relevant to what you see in the check, but I
    > installed scipy_distutils AFTER installing scipy.


    scipy_distutils is no longer used. All of its functionality has been folded into
    numpy.

    What do your environment variables look like when you build?

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #12
  13. James Stroud

    James Stroud Guest

    Robert Kern wrote:
    >> Sorry. To clarify, making libpython2.5a available in a
    >> $LD_LIBRARY_PATH was necessary to build numpy and scipy.

    >
    > I don't see how that could have affected anything. Static libraries
    > aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.


    OK--new thing learned for the day. <rant>I try things until a build
    works. I consider everything I do until acquiring a working build as
    necessary. Going back to see exactly what I did to make things work is
    not a viable option given that every new program I want requires my
    building it and all of its dependencies--a consequence of the [adjective
    deleted] beuracratic constraints (with regard to the machine on my
    desktop) that have evolved here. These constraints are no doubt a
    byproduct of the "Institute" part of my signature below, associated
    paranoia, and over-worked sys-admins.</rant>

    The upside is that my entire environment turns out to be roughly
    transportable to other Linux machines on the network via an NFS home
    directory--more or less independent of the configuration of said other
    machines.

    > What do your environment variables look like when you build?


    euler 2% echo $CPPFLAGS
    -I/data10/users/jstroud/Programs/include
    -I/data10/users/jstroud/Programs/qt/include -I/usr/include
    -I/data10/users/jstroud/Programs/include/freetype2/freetype
    euler 3% echo $LD_LIBRARY_PATH
    /data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
    euler 4% echo $LDFLAGS
    -L/data10/users/jstroud/Programs/lib
    -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3

    Here is how I customize my ld.so.cache:

    euler 9% cat ldconf
    rm ld.so.cache
    /sbin/ldconfig -v -f ./ld.so.conf -C ./ld.so.cache
    echo source this

    euler 10% cat ld.so.conf
    /data10/users/jstroud/Programs/lib
    /data10/users/jstroud/Programs/qt/lib
    /data10/users/jstroud/Programs/newlib
    /usr/lib
    /usr/local/lib
    /usr/X11R6/lib
    /usr/lib/qt3/lib
    /usr/lib/wine
    /usr/include/pgsql

    Again, see the above rant for an explanation about any needless paths or
    flags--though advice and knowledge is welcome.

    James

    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #13
  14. James Stroud

    James Stroud Guest

    Robert Kern wrote:
    > James Stroud wrote:
    >
    >> Well I added those arguments to $LDFLAGS, but it seemed like a pretty
    >> miserable hack, especially the "-shared" part.

    >
    >
    > Hmmm. Did you have $LDFLAGS set to anything (or more likely, nothing)
    > before? I.e. did you have something like this?
    >
    > export LDFLAGS=""
    > python setup.py build
    >
    > If such environment variables are present, I think that numpy.distutils
    > may clobber the values that are inferred from the configuration.
    >


    Didn't see this message last night.

    No, my $LDFLAGS were set. I haven't tried unsetenv'ing them (I use tcsh
    as a consequence of my anti-bash religious upbringing).

    James

    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #14
  15. James Stroud schrieb:
    >> What happens if you omit these flags?

    >
    > Please see my last message to Robert Kern.


    If you don't want me to help you, that's fine, then I won't.

    I couldn't find an answer to this question in any of your
    messages in this thread.

    Still, it *should* work out of the box. If it doesn't, it
    means there is an error somewhere. If you don't want to
    help finding the error, that's fine, too (but please say
    so, because that will constrain the solution space).

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Sep 22, 2006
    #15
  16. James Stroud

    James Stroud Guest

    Martin v. Löwis wrote:
    > James Stroud schrieb:
    >
    >>>What happens if you omit these flags?

    >>
    >>Please see my last message to Robert Kern.

    >
    >
    > If you don't want me to help you, that's fine, then I won't.
    >
    > I couldn't find an answer to this question in any of your
    > messages in this thread.


    Sorry, I didn't mean to sound dismissive. Perhaps I misunderstood what
    "these" actually referred to. The results of omitting these flags
    specifically:

    "-lpython2.5 -lpthread -lm -lutil -ldl -shared"

    Are posted here for both numpy and scipy:

    http://www.jamesstroud.com/build-scipy-logs/build-scipy-py2.5-euler.txt
    http://www.jamesstroud.com/build-scipy-logs/build-numpy-py2.5-euler.txt

    I have not tried unsetenv'ing $LDFLAGS entirely, an option which did not
    occur to me given their necessity for just about everything else. If you
    think that this is a worthwhile experiment, I will run it.


    > Still, it *should* work out of the box. If it doesn't, it
    > means there is an error somewhere. If you don't want to
    > help finding the error, that's fine, too (but please say
    > so, because that will constrain the solution space).


    I think I would like to know how to avoid or correct these sort of
    issues in the future, which seem to be limited, for me at least, to
    scipy and numpy, with the possible exception of MySQLdb and its
    dependency on zlib. Ideally, I would like to understand exactly what
    causes these errors in addition to knowing what I can do to correct them
    in the future.


    James


    --
    James Stroud
    UCLA-DOE Institute for Genomics and Proteomics
    Box 951570
    Los Angeles, CA 90095

    http://www.jamesstroud.com/
     
    James Stroud, Sep 22, 2006
    #16
  17. James Stroud

    Robert Kern Guest

    James Stroud wrote:
    > Robert Kern wrote:
    >>> Sorry. To clarify, making libpython2.5a available in a
    >>> $LD_LIBRARY_PATH was necessary to build numpy and scipy.

    >> I don't see how that could have affected anything. Static libraries
    >> aren't looked up in $LD_LIBRARY_PATH. Certainly not for build-time linking.

    >
    > OK--new thing learned for the day. <rant>I try things until a build
    > works. I consider everything I do until acquiring a working build as
    > necessary. Going back to see exactly what I did to make things work is
    > not a viable option given that every new program I want requires my
    > building it and all of its dependencies--a consequence of the [adjective
    > deleted] beuracratic constraints (with regard to the machine on my
    > desktop) that have evolved here. These constraints are no doubt a
    > byproduct of the "Institute" part of my signature below, associated
    > paranoia, and over-worked sys-admins.</rant>


    :) I've always thought that a Kubuntu LiveCD would be a useful path around much
    of that nonsense, but I've fortunately not been forced to attempt it.

    >> What do your environment variables look like when you build?

    >
    > euler 2% echo $CPPFLAGS
    > -I/data10/users/jstroud/Programs/include
    > -I/data10/users/jstroud/Programs/qt/include -I/usr/include
    > -I/data10/users/jstroud/Programs/include/freetype2/freetype
    > euler 3% echo $LD_LIBRARY_PATH
    > /data10/users/jstroud/Programs/lib:/data10/users/jstroud/Programs/qt/lib:/usr/lib:/usr/X11R6/lib/:/usr/local/lib:/usr/lib/gcc-lib/i386-redhat-linux/3.2.3:/data10/users/jstroud/Programs/ccp4-5.0.2/lib
    > euler 4% echo $LDFLAGS
    > -L/data10/users/jstroud/Programs/lib
    > -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3


    Okay, this is possibly part of the problem. numpy.distutils handles FORTRAN code
    differently than other extension modules; this is why pure C extension modules
    both inside numpy and elsewehere were linking fine. Try unsetenving $CPPFLAGS
    and $LDFLAGS and adding those arguments to build_ext, if necessary.

    Looking on line 516 of numpy/fcompiler/__init__.py (why there's a class
    definition in __init__.py, I'll never know), it does look like the linker flags
    which are carefully constructed by numpy.distutils are indeed overridden by the
    user's setting of $LDFLAGS.

    The fact that this appears to be incompatible with how distutils does it for
    everything else is a bug, IMO, although it *is* a useful feature if you really
    do need to completely control the linker flags. That's probably what the
    implementor was thinking. Although, sadly, not documenting. He should be back
    from his honeymoon, now, so I'll raise the issue with him.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Sep 22, 2006
    #17
  18. James Stroud

    Ben Finney Guest

    James Stroud <> writes:

    > <rant>I try things until a build works. I consider everything I do
    > until acquiring a working build as necessary. Going back to see
    > exactly what I did to make things work is not a viable option


    Before trying such "I don't know what I need to do, but I'm going to
    want to know after I do it" procedures, I usually start 'script' in
    the shell session and then start all my experimenting. ('script' is a
    tool from BSD available on most GNU/Linux distributions as well.)

    That way, the entire session, fumbling dead-ends and all, is available
    for later examination and trimming back to a "what I did to get it
    working" document.

    --
    \ "I have yet to see any problem, however complicated, which, |
    `\ when you looked at it in the right way, did not become still |
    _o__) more complicated." -- Paul Anderson |
    Ben Finney
     
    Ben Finney, Sep 23, 2006
    #18
  19. James Stroud schrieb:
    > I think I would like to know how to avoid or correct these sort of
    > issues in the future, which seem to be limited, for me at least, to
    > scipy and numpy, with the possible exception of MySQLdb and its
    > dependency on zlib. Ideally, I would like to understand exactly what
    > causes these errors in addition to knowing what I can do to correct them
    > in the future.


    Let's take a specific failure, namely the line

    /auto_nfs/data10/users/jstroud/Programs/bin/g77
    -L/data10/users/jstroud/Programs/lib
    -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
    build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
    -L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
    build/lib.linux-i686-2.5/numpy/core/_dotblas.so

    This gives errors like

    build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o(.text+0x758):numpy/core/blasdot/_dotblas.c:226:
    undefined reference to `PyTuple_New'

    That's not too surprising: this line tries to link the input
    *as an executable program*, despite calling the output
    _dotblas.so. In an executable program, all symbols need to
    be defined; that's why it it complains about so many missing
    symbols (including MAIN__ - which should never be missing
    in a library). Even though adding python2.5.a to the linker
    link makes these symbols appear, the result still won't
    work, as you can't use an executable program file as if
    it were a shared library.

    Now, compare this to a succeeding build of a C extension
    module,

    gcc -pthread -shared -L/data10/users/jstroud/Programs/lib
    -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
    -I/data10/users/jstroud/Programs/include
    -I/data10/users/jstroud/Programs/qt/include -I/usr/include
    -I/data10/users/jstroud/Programs/include/freetype2/freetype
    build/temp.linux-i686-2.5/build/src.linux-i686-2.5/numpy/core/src/umathmodule.o
    -lm -o build/lib.linux-i686-2.5/numpy/core/umath.so

    Notice that this passes -shared to the compiler, requesting
    construction of a shared library. This is the problem with
    the g77 linker line; try invoking

    /auto_nfs/data10/users/jstroud/Programs/bin/g77 -shared
    -L/data10/users/jstroud/Programs/lib
    -L/usr/lib/gcc-lib/i386-redhat-linux/3.2.3
    build/temp.linux-i686-2.5/numpy/core/blasdot/_dotblas.o
    -L/data10/users/jstroud/Programs/lib -lblas -lg2c -o
    build/lib.linux-i686-2.5/numpy/core/_dotblas.so

    manually (whether or not -pthread is also necessary
    or supported for g77, I don't know). This should at least
    make the complaints about missing symbols go away; you
    should then see whether the resulting module can be
    imported in Python.

    If that solves the problem, the question is why the
    -shared option isn't passed automatically; your setting
    LDFLAGS might indeed be a cause.

    Regards,
    Martin
     
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=, Sep 23, 2006
    #19
  20. James Stroud

    James Stroud Guest

    Ben Finney wrote:
    > James Stroud <> writes:
    >
    >> <rant>I try things until a build works. I consider everything I do
    >> until acquiring a working build as necessary. Going back to see
    >> exactly what I did to make things work is not a viable option

    >
    > Before trying such "I don't know what I need to do, but I'm going to
    > want to know after I do it" procedures, I usually start 'script' in
    > the shell session and then start all my experimenting. ('script' is a
    > tool from BSD available on most GNU/Linux distributions as well.)
    >
    > That way, the entire session, fumbling dead-ends and all, is available
    > for later examination and trimming back to a "what I did to get it
    > working" document.
    >


    Beautiful suggestion. I've never even heard of that before. With OS X
    Terminal.app, I can buffer "unlimited lines" on my 6 year old ibook I
    use at home and on the road. But at work, on my brand new dual pentium,
    I've found myself limited to the thousand or so lines that konsole gives me.

    James
     
    James Stroud, Sep 23, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?V2lsbGlhbSBTdWxsaXZhbg==?=

    vs2005 publish website doing bad things, bad things

    =?Utf-8?B?V2lsbGlhbSBTdWxsaXZhbg==?=, Oct 25, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    626
    =?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=
    Oct 25, 2006
  2. Chris Shenton
    Replies:
    1
    Views:
    1,491
    =?ISO-8859-1?Q?Michael_Str=F6der?=
    Aug 24, 2007
  3. Benedict Verheyen
    Replies:
    0
    Views:
    205
    Benedict Verheyen
    Jan 20, 2012
  4. Tom Copeland
    Replies:
    0
    Views:
    107
    Tom Copeland
    Mar 10, 2005
  5. Intransition

    [ANN] Ruby Setup 5 (setup.rb)

    Intransition, Jan 13, 2010, in forum: Ruby
    Replies:
    0
    Views:
    444
    Intransition
    Jan 13, 2010
Loading...

Share This Page