python extensions: including project local headers

Discussion in 'Python' started by J Kenneth King, Oct 23, 2008.

  1. Hey everyone,

    I'm working on a python extension wrapper around Rob Hess'
    implementation of a SIFT feature detector. I'm working on a
    computer-vision based project that requires interfacing with Python at
    the higher layers, so I figured the best way to handle this would be in
    C (since my initial implementation in python was ungodly and slow).

    I can get distutils to compile the extension and install it in the
    python path, but when I go to import it I get the wonderful exception:

    ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    symbol: _sift_features

    Of course, _sift_features is a function defined in his header that I'm
    #including in my extension.

    His sources are sitting in my project root under sift/ while my source
    is under src/ -- My setup.py is as follows:

    Code:
    
    from distutils.core import setup, Extension
    
    pysift = Extension('pysift',
                       include_dirs = ['sift/include'],
                       sources = ['src/pysift.c'],
                       extra_link_args = ['-lm', '-lcv', '-lcxcore',
                                          '-lhighgui', '-lcvaux'])
    
    setup(name = 'pysift',
          version = '0.0',
          description = 'A SIFT feature detection package',
          author = 'James Kenneth King',
          author_email = "",
          url = "http://agentultra.com/",
          long_description = """
          A python extension package for detecting SIFT
          features using Rob Hess' C implementation.
    
          http://web.engr.oregonstate.edu/~hess/
    
          Original SIFT feature descriptors by David Lowe
          and patented by the University of British Columbia.
          """,
          ext_modules = [pysift])
    
    
    And the include to Rob's header file is on the second line of pysift.c:

    #include "sift.h"

    The weird thing (to me in my somewhat hackish knowledge of C) is that I
    can use all the #defines from sift.h with no complaints from the
    preprocessor (in fact, there are no complaints at all from the compiler
    when compiling the extension module).

    Once I get this bugger working, I'll be setting up a project page to
    share sources and will also be releasing extension wrappers to the
    OpenCV libraries.

    I've never released any code before so any help getting this right and
    proper for the community would be greatly appreciated.

    Cheers.
     
    J Kenneth King, Oct 23, 2008
    #1
    1. Advertising

  2. On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:

    >
    > Hey everyone,
    >
    > I'm working on a python extension wrapper around Rob Hess'
    > implementation of a SIFT feature detector. I'm working on a
    > computer-vision based project that requires interfacing with Python at
    > the higher layers, so I figured the best way to handle this would be
    > in
    > C (since my initial implementation in python was ungodly and slow).
    >
    > I can get distutils to compile the extension and install it in the
    > python path, but when I go to import it I get the wonderful exception:
    >
    > ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    > symbol: _sift_features



    Kenneth,
    You're close but not interpreting the error quite correctly. This
    isn't an error from the compiler or preprocessor, it's a library
    error. Assuming this is dynamically linked, your OS is reporting that,
    at runtime, it can't find the library that contains _sift_features.
    Make sure that it's somewhere where your OS can find it.

    HTH
    Philip
     
    Philip Semanchuk, Oct 23, 2008
    #2
    1. Advertising

  3. J Kenneth King

    Robert Kern Guest

    Philip Semanchuk wrote:
    >
    > On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >
    >>
    >> Hey everyone,
    >>
    >> I'm working on a python extension wrapper around Rob Hess'
    >> implementation of a SIFT feature detector. I'm working on a
    >> computer-vision based project that requires interfacing with Python at
    >> the higher layers, so I figured the best way to handle this would be in
    >> C (since my initial implementation in python was ungodly and slow).
    >>
    >> I can get distutils to compile the extension and install it in the
    >> python path, but when I go to import it I get the wonderful exception:
    >>
    >> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >> symbol: _sift_features

    >
    >
    > Kenneth,
    > You're close but not interpreting the error quite correctly. This isn't
    > an error from the compiler or preprocessor, it's a library error.
    > Assuming this is dynamically linked, your OS is reporting that, at
    > runtime, it can't find the library that contains _sift_features. Make
    > sure that it's somewhere where your OS can find it.


    It looks like the library implementing it was not linked into the extension.
    sift_features() is not part of OpenCV.

    James, are you including the source of Rob Hess's implementation with your
    extension, or are you trying to link against an already installed version of the
    library? If the former, you need to add the C sources to the pysift Extension().
    If the latter, you need to add the name of the library to the list of libraries.

    Also, you don't want to pass the list of libraries with extra_link_args.
    Instead, use libraries=.

    pysift = Extension('pysift',
    include_dirs = ['sift/include'],
    sources = ['src/pysift.c'],
    libraries = ['feat', 'cv', 'cxcore', 'highgui',
    'cvaux', 'm'])

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Oct 23, 2008
    #3
  4. Robert Kern <> writes:

    > Philip Semanchuk wrote:
    >>
    >> On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >>
    >>>
    >>> Hey everyone,
    >>>
    >>> I'm working on a python extension wrapper around Rob Hess'
    >>> implementation of a SIFT feature detector. I'm working on a
    >>> computer-vision based project that requires interfacing with Python at
    >>> the higher layers, so I figured the best way to handle this would be in
    >>> C (since my initial implementation in python was ungodly and slow).
    >>>
    >>> I can get distutils to compile the extension and install it in the
    >>> python path, but when I go to import it I get the wonderful exception:
    >>>
    >>> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >>> symbol: _sift_features

    >>
    >>
    >> Kenneth,
    >> You're close but not interpreting the error quite correctly. This
    >> isn't an error from the compiler or preprocessor, it's a library
    >> error. Assuming this is dynamically linked, your OS is reporting
    >> that, at runtime, it can't find the library that contains
    >> _sift_features. Make sure that it's somewhere where your OS can find
    >> it.

    >
    > It looks like the library implementing it was not linked into the
    > extension. sift_features() is not part of OpenCV.
    >
    > James, are you including the source of Rob Hess's implementation with
    > your extension, or are you trying to link against an already installed
    > version of the library? If the former, you need to add the C sources
    > to the pysift Extension(). If the latter, you need to add the name of
    > the library to the list of libraries.


    I'm including Rob Hess' sources with the extension.

    Would that mean I should add library_dirs to Extension() to point to the
    sources in the project's path?

    > Also, you don't want to pass the list of libraries with
    > extra_link_args. Instead, use libraries=.
    >
    > pysift = Extension('pysift',
    > include_dirs = ['sift/include'],
    > sources = ['src/pysift.c'],
    > libraries = ['feat', 'cv', 'cxcore', 'highgui',
    > 'cvaux', 'm'])


    Thanks for taking a moment to help me out. :)
     
    J Kenneth King, Oct 23, 2008
    #4
  5. J Kenneth King

    Robert Kern Guest

    J Kenneth King wrote:
    > Robert Kern <> writes:
    >
    >> Philip Semanchuk wrote:
    >>> On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >>>
    >>>> Hey everyone,
    >>>>
    >>>> I'm working on a python extension wrapper around Rob Hess'
    >>>> implementation of a SIFT feature detector. I'm working on a
    >>>> computer-vision based project that requires interfacing with Python at
    >>>> the higher layers, so I figured the best way to handle this would be in
    >>>> C (since my initial implementation in python was ungodly and slow).
    >>>>
    >>>> I can get distutils to compile the extension and install it in the
    >>>> python path, but when I go to import it I get the wonderful exception:
    >>>>
    >>>> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >>>> symbol: _sift_features
    >>>
    >>> Kenneth,
    >>> You're close but not interpreting the error quite correctly. This
    >>> isn't an error from the compiler or preprocessor, it's a library
    >>> error. Assuming this is dynamically linked, your OS is reporting
    >>> that, at runtime, it can't find the library that contains
    >>> _sift_features. Make sure that it's somewhere where your OS can find
    >>> it.

    >> It looks like the library implementing it was not linked into the
    >> extension. sift_features() is not part of OpenCV.
    >>
    >> James, are you including the source of Rob Hess's implementation with
    >> your extension, or are you trying to link against an already installed
    >> version of the library? If the former, you need to add the C sources
    >> to the pysift Extension(). If the latter, you need to add the name of
    >> the library to the list of libraries.

    >
    > I'm including Rob Hess' sources with the extension.
    >
    > Would that mean I should add library_dirs to Extension() to point to the
    > sources in the project's path?


    No, you would add the source file names to the sources= list.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Oct 23, 2008
    #5
  6. Philip Semanchuk <> writes:

    > On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >
    >>
    >> Hey everyone,
    >>
    >> I'm working on a python extension wrapper around Rob Hess'
    >> implementation of a SIFT feature detector. I'm working on a
    >> computer-vision based project that requires interfacing with Python at
    >> the higher layers, so I figured the best way to handle this would be
    >> in
    >> C (since my initial implementation in python was ungodly and slow).
    >>
    >> I can get distutils to compile the extension and install it in the
    >> python path, but when I go to import it I get the wonderful exception:
    >>
    >> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >> symbol: _sift_features

    >
    >
    > Kenneth,
    > You're close but not interpreting the error quite correctly. This
    > isn't an error from the compiler or preprocessor, it's a library
    > error. Assuming this is dynamically linked, your OS is reporting that,
    > at runtime, it can't find the library that contains _sift_features.
    > Make sure that it's somewhere where your OS can find it.


    This is basically what I was looking for help with. So far the project
    directory is:

    /pysift
    /sift
    ..
    /include
    ..
    sift.h
    /src
    ..
    sift.c
    /src
    pysift.c
    setup.py

    I thought I could just #include "sift.h" in pysift.c as long as
    distutils passed the right -I path to gcc.

    Maybe I should compile the sift code as a shared object and link it to
    my extension? How would I get distutils to build the makefile and tell
    gcc how to link it?

    Thanks for the reply. Python has spoiled me and my C is rather
    rusty. :)
     
    J Kenneth King, Oct 23, 2008
    #6
  7. On Oct 23, 2008, at 3:18 PM, J Kenneth King wrote:

    > Philip Semanchuk <> writes:
    >
    >> On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >>
    >>>
    >>> Hey everyone,
    >>>
    >>> I'm working on a python extension wrapper around Rob Hess'
    >>> implementation of a SIFT feature detector. I'm working on a
    >>> computer-vision based project that requires interfacing with
    >>> Python at
    >>> the higher layers, so I figured the best way to handle this would be
    >>> in
    >>> C (since my initial implementation in python was ungodly and slow).
    >>>
    >>> I can get distutils to compile the extension and install it in the
    >>> python path, but when I go to import it I get the wonderful
    >>> exception:
    >>>
    >>> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >>> symbol: _sift_features

    >>
    >>
    >> Kenneth,
    >> You're close but not interpreting the error quite correctly. This
    >> isn't an error from the compiler or preprocessor, it's a library
    >> error. Assuming this is dynamically linked, your OS is reporting
    >> that,
    >> at runtime, it can't find the library that contains _sift_features.
    >> Make sure that it's somewhere where your OS can find it.

    >
    > This is basically what I was looking for help with. So far the project
    > directory is:
    >
    > /pysift
    > /sift
    > ..
    > /include
    > ..
    > sift.h
    > /src
    > ..
    > sift.c
    > /src
    > pysift.c
    > setup.py
    >
    > I thought I could just #include "sift.h" in pysift.c as long as
    > distutils passed the right -I path to gcc.


    That's true, and it sounds like you've got that part working.


    > Maybe I should compile the sift code as a shared object and link it to
    > my extension? How would I get distutils to build the makefile and tell
    > gcc how to link it?
    >
    > Thanks for the reply. Python has spoiled me and my C is rather
    > rusty. :)


    I don't know how to get setup.py to build a shared object separately.
    I am in the same Python/C situation as you. I'm scrubbing the rust off
    of my C skills and I'm also a n00b at developing extensions. I've
    learned a lot from looking at other people's setup code, so maybe I
    can help you there.

    My posix_ipc module links to the realtime lib "rt" and here's the
    relevant snippets of setup.py:

    ------------------------------
    import distutils.core as duc

    libraries = [ ]

    libraries.append("rt")

    source_files = ["posix_ipc_module.c"]

    ext_modules = [ duc.Extension("posix_ipc",
    source_files,
    define_macros=define_macros,
    libraries=libraries
    )
    ]

    duc.setup(name="posix_ipc", version=VERSION, ext_modules=ext_modules)

    ------------------------------

    You can download the whole thing here if you want to examine all the
    code:
    http://semanchuk.com/philip/posix_ipc/

    HTH
    Philip
     
    Philip Semanchuk, Oct 23, 2008
    #7
  8. J Kenneth King

    Robert Kern Guest

    J Kenneth King wrote:
    > Philip Semanchuk <> writes:
    >
    >> On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >>
    >>> Hey everyone,
    >>>
    >>> I'm working on a python extension wrapper around Rob Hess'
    >>> implementation of a SIFT feature detector. I'm working on a
    >>> computer-vision based project that requires interfacing with Python at
    >>> the higher layers, so I figured the best way to handle this would be
    >>> in
    >>> C (since my initial implementation in python was ungodly and slow).
    >>>
    >>> I can get distutils to compile the extension and install it in the
    >>> python path, but when I go to import it I get the wonderful exception:
    >>>
    >>> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >>> symbol: _sift_features

    >>
    >> Kenneth,
    >> You're close but not interpreting the error quite correctly. This
    >> isn't an error from the compiler or preprocessor, it's a library
    >> error. Assuming this is dynamically linked, your OS is reporting that,
    >> at runtime, it can't find the library that contains _sift_features.
    >> Make sure that it's somewhere where your OS can find it.

    >
    > This is basically what I was looking for help with. So far the project
    > directory is:
    >
    > /pysift
    > /sift
    > ..
    > /include
    > ..
    > sift.h
    > /src
    > ..
    > sift.c
    > /src
    > pysift.c
    > setup.py
    >
    > I thought I could just #include "sift.h" in pysift.c as long as
    > distutils passed the right -I path to gcc.
    >
    > Maybe I should compile the sift code as a shared object and link it to
    > my extension? How would I get distutils to build the makefile and tell
    > gcc how to link it?


    I don't recommend doing that, if you can avoid it. distutils does not really
    support that. If you can get the sift files to compile under distutils as part
    of the Extension, that is by far the best option.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Oct 23, 2008
    #8
  9. Philip Semanchuk <> writes:

    > On Oct 23, 2008, at 3:18 PM, J Kenneth King wrote:
    >
    >> Philip Semanchuk <> writes:
    >>
    >>> On Oct 23, 2008, at 11:36 AM, J Kenneth King wrote:
    >>>
    >>>>
    >>>> Hey everyone,
    >>>>
    >>>> I'm working on a python extension wrapper around Rob Hess'
    >>>> implementation of a SIFT feature detector. I'm working on a
    >>>> computer-vision based project that requires interfacing with
    >>>> Python at
    >>>> the higher layers, so I figured the best way to handle this would be
    >>>> in
    >>>> C (since my initial implementation in python was ungodly and slow).
    >>>>
    >>>> I can get distutils to compile the extension and install it in the
    >>>> python path, but when I go to import it I get the wonderful
    >>>> exception:
    >>>>
    >>>> ImportError: /usr/lib/python2.5/site-packages/pysift.so: undefined
    >>>> symbol: _sift_features
    >>>
    >>>
    >>> Kenneth,
    >>> You're close but not interpreting the error quite correctly. This
    >>> isn't an error from the compiler or preprocessor, it's a library
    >>> error. Assuming this is dynamically linked, your OS is reporting
    >>> that,
    >>> at runtime, it can't find the library that contains _sift_features.
    >>> Make sure that it's somewhere where your OS can find it.

    >>
    >> This is basically what I was looking for help with. So far the project
    >> directory is:
    >>
    >> /pysift
    >> /sift
    >> ..
    >> /include
    >> ..
    >> sift.h
    >> /src
    >> ..
    >> sift.c
    >> /src
    >> pysift.c
    >> setup.py
    >>
    >> I thought I could just #include "sift.h" in pysift.c as long as
    >> distutils passed the right -I path to gcc.

    >
    > That's true, and it sounds like you've got that part working.
    >
    >
    >> Maybe I should compile the sift code as a shared object and link it to
    >> my extension? How would I get distutils to build the makefile and tell
    >> gcc how to link it?
    >>
    >> Thanks for the reply. Python has spoiled me and my C is rather
    >> rusty. :)

    >
    > I don't know how to get setup.py to build a shared object separately.
    > I am in the same Python/C situation as you. I'm scrubbing the rust off
    > of my C skills and I'm also a n00b at developing extensions. I've
    > learned a lot from looking at other people's setup code, so maybe I
    > can help you there.
    >
    > My posix_ipc module links to the realtime lib "rt" and here's the
    > relevant snippets of setup.py:
    >
    > ------------------------------
    > import distutils.core as duc
    >
    > libraries = [ ]
    >
    > libraries.append("rt")
    >
    > source_files = ["posix_ipc_module.c"]
    >
    > ext_modules = [ duc.Extension("posix_ipc",
    > source_files,
    > define_macros=define_macros,
    > libraries=libraries
    > )
    > ]
    >
    > duc.setup(name="posix_ipc", version=VERSION, ext_modules=ext_modules)
    >
    > ------------------------------
    >
    > You can download the whole thing here if you want to examine all the
    > code:
    > http://semanchuk.com/philip/posix_ipc/
    >
    > HTH
    > Philip


    I'll take a look, thanks! :)
     
    J Kenneth King, Oct 24, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Miguel

    Including headers only once

    Miguel, Oct 30, 2003, in forum: C Programming
    Replies:
    3
    Views:
    324
  2. Tim Fuehner
    Replies:
    0
    Views:
    409
    Tim Fuehner
    Dec 8, 2003
  3. Replies:
    4
    Views:
    504
    red floyd
    Jan 19, 2007
  4. Shanti Braford

    Including core class extensions from gems

    Shanti Braford, Jan 5, 2007, in forum: Ruby
    Replies:
    1
    Views:
    105
    Eric Hodel
    Jan 5, 2007
  5. Nu
    Replies:
    11
    Views:
    166
    John Bokma
    Oct 31, 2006
Loading...

Share This Page