calling Pyrex results from C

Discussion in 'Python' started by Kyler Laird, Jan 20, 2004.

  1. Kyler Laird

    Kyler Laird Guest

    I need to submit C/C++ code for a class. (It's not a programming
    class. The choice of language is inertial. I think that it mostly
    serves to distract students from the course subject.) I'm fairly
    fluent with C but it hurts to think about writing in C when Python
    is *so* much more appropriate for these operations.

    I'd like to keep my sanity and satisfy the course requirements by
    programming in Python and converting the code to C. It looks like
    a few people have scratched this itch already, but of the
    translators I found, most of them (Python2C, P2C, PyFront) seem to
    be dead. Pyrex, however, appears to be well-maintained and is even
    available as a Debian package.
    http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

    I realize that the goal of Pyrex is to create a module that can be
    called from Python. For my need of calling the result from C, the
    other utilities are probably more appropriate but I think that
    Pyrex could be useful to me for less contrived situations than this
    so it's worthwhile to learn more about it.


    http://www.cosc.canterbury.ac.nz/~g...ion/Doc/overview.html#InterfacingWithExternal

    You can also use public declarations to make C functions
    and variables defined in a Pyrex module available to
    external C code. The need for this is expected to be less
    frequent, but you might want to do it, for example, if you
    are embedding Python in another application as a scripting
    language. Just as a Pyrex module can be used as a bridge to
    allow Python code to call C code, it can also be used to
    allow C code to call Python code.

    [...]

    You can make C variables and functions defined in a Pyrex
    module accessible to external C code (or another Pyrex
    module) using the public keyword

    I've discovered that as long as everything in the code is cdef-ed,
    I can call Pyrex code from my C code. If, however, I so much as
    mention anything Pythonish, it segfaults during execution.

    For example, this is fine.
    cdef public int foo():
    cdef int i
    i = 123

    return(i)

    And this results in a segfault.
    cdef public int foo():
    cdef int i
    i = 123
    j = 5

    return(i)

    This means that I can't, for example, make a function that takes a
    filename (string), uses PIL to do a bunch of operations, and
    returns a string.

    Any suggestions (besides "suck it up and do it all in C")?

    Thank you.

    --kyler
     
    Kyler Laird, Jan 20, 2004
    #1
    1. Advertising

  2. [Kyler Laird]

    > I realize that the goal of Pyrex is to create a module that can be
    > called from Python.


    Or vice-versa. I found Pyrex useful for calling Python modules from
    within C programs which were not Python-aware to start with. You may
    even write wholly Pyrex programs.

    > I've discovered that as long as everything in the code is cdef-ed,
    > I can call Pyrex code from my C code. If, however, I so much as
    > mention anything Pythonish, it segfaults during execution.


    There is some magic needed to establish Python context, not much.
    Hoping that it will help, here is a template I use when generating
    Pyrex main programs. `%(module)s' shall be replaced by the Pyrex module
    name, as Pyrex needs to initialise itself. When Python calls Pyrex,
    initialisation occurs automatically at `import' time, but when C
    initially calls Pyrex, initialisation should be explicitly launched.



    cdef extern void Py_Initialize()
    cdef extern void Py_Finalize()
    cdef extern void init%(module)s()

    cdef extern int main(int argc, char **argv):
    cdef int index, status
    Py_Initialize()
    init%(module)s()
    import sys
    try:
    arguments = []
    for index from 0 <= index < argc:
    arguments.append(argv[index])
    sys.argv = arguments
    result = main_application()
    raise SystemExit, result or 0
    except SystemExit, result:
    result = str(result)
    try:
    status = int(result)
    except ValueError:
    if not result.endswith('\n'):
    result = result + '\n'
    sys.stderr.write(result)
    status = 1
    except:
    import traceback
    traceback.print_exc()
    status = 1
    Py_Finalize()
    return status


    --
    Fran├žois Pinard http://www.iro.umontreal.ca/~pinard
     
    =?iso-8859-1?Q?Fran=E7ois?= Pinard, Jan 20, 2004
    #2
    1. Advertising

  3. Kyler Laird

    John J. Lee Guest

    Kyler Laird <> writes:
    [...]
    > This means that I can't, for example, make a function that takes a
    > filename (string), uses PIL to do a bunch of operations, and
    > returns a string.


    I don't see why you shouldn't be able to write a module in Pyrex, then
    use standard embedding techniques to embed Python and then call your
    module with a few C API calls. Or do the same, but also write a Pyrex
    cdef function to wrap it all up, so you can just call that as a "plain
    old" C function instead of those final few C API calls (you'll still
    need other embedding calls, of course, to initialise Python, etc.).

    Try the pyrex mailing list if you're stuck.

    I'd very definitely check to make sure you aren't going to get marked
    down for being a smartass, though (even if you're generating C that
    doesn't depend on Python at all, it might still be unreadable).


    > Any suggestions (besides "suck it up and do it all in C")?


    I think I'd do it in C, regardless of whether or not it's allowable to
    use Pyrex.


    John
     
    John J. Lee, Jan 20, 2004
    #3
  4. Kyler Laird

    Kyler Laird Guest

    On Tue, Jan 20, 2004 at 12:27:33PM -0500, Fran?ois Pinard wrote:

    > There is some magic needed to establish Python context, not much.
    > Hoping that it will help, here is a template I use when generating
    > Pyrex main programs.


    Paul started me on this path and you've given me exactly what I need to
    keep going.

    Thank you!

    --kyler
     
    Kyler Laird, Jan 20, 2004
    #4
  5. Kyler Laird

    Kyler Laird Guest

    Paul Prescod graciously responded to my plea for help by gently asking
    if I had initialized Python. Doh! I had gotten lazy and assumed that
    Pyrex would take care of such details. (How could it?!)

    Initializing the interpreter seems to have the simple test case I made
    working smoothly. Now it's on to bigger and better things...

    Pyrex is *so* cool.

    Thank you, Paul!

    --kyler
     
    Kyler Laird, Jan 21, 2004
    #5
  6. [Kyler Laird]
    > On Tue, Jan 20, 2004 at 12:27:33PM -0500, Fran?ois Pinard wrote:


    > > There is some magic needed to establish Python context, not much.
    > > Hoping that it will help, here is a template I use when generating
    > > Pyrex main programs.


    > Paul started me on this path and you've given me exactly what I need to
    > keep going.


    Good. :)

    There is also some linking magic, which differs from system to system,
    it might be documented in the Python Embedding API, I'm not sure. If
    you do not stumble on the proper recipe, I'll share what I use on Linux.
    Just ask if you want me to go search into some of my Makefiles :).

    --
    Fran├žois Pinard http://www.iro.umontreal.ca/~pinard
     
    =?iso-8859-1?Q?Fran=E7ois?= Pinard, Jan 21, 2004
    #6
  7. Kyler Laird

    Kyler Laird Guest

    The help I've gotten in this thread has enabled me to complete my
    assignment...well, at least I can generate C code that spits out
    the answer.

    I'd like to factor out more of the Python so that my handwritten
    C code can do more of the high-level operations. I'm having some
    difficulties passing data between my C and the Pyrex code.

    One of the first things I thought I'd build is a helper function
    to return a string representation of an object. I thought that
    this would be fairly straightforward. Unfortunately it segfaults
    when called with some Python objects.

    Pyrex:
    cdef public image_file_open(char* image_filename):
    return(Image.open(image_filename))

    cdef public image_size(image_PIL):
    return(Image.size)

    cdef public char* string(x):
    s = str(x)
    return(s)

    my C:
    void *im, *im_size;
    im = image_file_open(input_filename);
    im_size = image_size(im);
    printf("im=%s\n", string(im));
    printf("im_size=%s\n", string(im_size));

    The first call to string() succeeds but the second one fails. I
    suspect that I've expected too much of Pyrex again. Do I need to
    allocate memory, manipulate reference counts or something like
    that?

    It's difficult for me to get back to thinking in terms of
    passing around data in C ways. I almost forgot what a boon
    tuples have been. (Perhaps I need to code in assembly for
    awhile to get back in that frame of mind.) Someday I might try
    to pass structures between C and Pyrex. I'm slowly getting
    comfortable with Pyrex but I can see how my code can get *much*
    cleaner once I understand more of its capabilities.

    Thank you.

    --kyler
     
    Kyler Laird, Jan 21, 2004
    #7
  8. Kyler Laird

    Paul Prescod Guest

    Kyler Laird wrote:

    >
    > Pyrex:
    > cdef public image_file_open(char* image_filename):
    > return(Image.open(image_filename))
    >
    > cdef public image_size(image_PIL):
    > return(Image.size)
    >
    > cdef public char* string(x):
    > s = str(x)
    > return(s)
    >
    > my C:
    > void *im, *im_size;
    > im = image_file_open(input_filename);
    > im_size = image_size(im);
    > printf("im=%s\n", string(im));
    > printf("im_size=%s\n", string(im_size));
    >
    > The first call to string() succeeds but the second one fails. I
    > suspect that I've expected too much of Pyrex again. Do I need to
    > allocate memory, manipulate reference counts or something like
    > that?


    First, I'd suggest that "integer" is a perfectly good type in both C and
    Pyrex so you shouldn't pass around Python objects representing integers.

    Second, you aren't checking the return codes of your functions and C has
    no exception handling, tracebacks, etc. image_size is probably returning
    0 because it is probably throwing an exception because you are asking
    for the size attribute of the Image class rather than the image_PIL object.

    You actually would have gotten a Pyrex error on your console if your
    function were defined to return "int" or "void" because Pyrex would KNOW
    that there's no way you are doing exception handling so it would try to
    compensate by printing exceptions to the console. But I wouldn't depend
    on that feature because it doesn't help for functions that really should
    return objects. Better to check your return values.

    Once I fix the Image/Image_PIL error your code runs okay on my computer.
    But you are walking on thin ice and may just be lucky. It is simply
    not possible to work with strings generated at runtime in C without
    worrying about memory allocation sometime. In this case I have the
    strong suspicion that the string() function is either creating and
    destroying a string object and then returning you a pointer to the dead
    object's internal memory buffer (bad news!) or simply losing a reference
    to the (still living) string object: still not a good thing. Eyeballing
    the code I believe the former is the issue. You could check for sure by
    adding some printf's to the generated code to look at __pyx_v_s->ob_refcnt.

    Why not let Python do the "print" rather than using "printf".

    Paul Prescod
     
    Paul Prescod, Jan 21, 2004
    #8
  9. Kyler Laird

    Kyler Laird Guest

    On Wed, Jan 21, 2004 at 08:23:53AM -0800, Paul Prescod wrote:

    > First, I'd suggest that "integer" is a perfectly good type in both C and
    > Pyrex so you shouldn't pass around Python objects representing integers.


    That would be my inclination too if I just had an integer to pass.
    Are you suggesting that I should pass tuples of integers as pointers
    to (an array of) integers?

    > Second, you aren't checking the return codes of your functions and C has
    > no exception handling, tracebacks, etc. image_size is probably returning
    > 0 because it is probably throwing an exception because you are asking
    > for the size attribute of the Image class rather than the image_PIL object.


    I'm trying to keep my examples simple so that it's easy to pick out the
    mistakes.

    > Once I fix the Image/Image_PIL error your code runs okay on my computer.


    Grrr...yup, that does it. Dumb mistake - the result of pulling code
    out of various places and gluing it together without checks.

    > But you are walking on thin ice and may just be lucky. It is simply
    > not possible to work with strings generated at runtime in C without
    > worrying about memory allocation sometime. In this case I have the
    > strong suspicion that the string() function is either creating and
    > destroying a string object and then returning you a pointer to the dead
    > object's internal memory buffer (bad news!) or simply losing a reference
    > to the (still living) string object: still not a good thing. Eyeballing
    > the code I believe the former is the issue. You could check for sure by
    > adding some printf's to the generated code to look at __pyx_v_s->ob_refcnt.


    I feared this was a problem. I'll play with it.

    > Why not let Python do the "print" rather than using "printf".


    Printing is not always my final goal.

    Thank you.

    --kyler
     
    Kyler Laird, Jan 21, 2004
    #9
  10. Kyler Laird

    Kyler Laird Guest

    Paul Prescod <> writes:

    > 1. How is your teacher going to react when you hand in obfuscated-ish
    > C code that depends on the whole Python interpreter _and_ PIL?


    Shhh...don't tell anyone.
    Grades for ECE661 as of Wed Feb 4 15:01:35 2004
    Name hw1 hw2
    ---- ----- -----
    Maximums: 100.0 100.0
    Laird, Kyler 100.0 100.0
    Class Average: 93.8 86.6
    Class Standard Deviation: 20.1 24.3

    This is Computer Vision, not Introduction to Pointers and Memory Allocation.
    Pyrex frees me to concentrate on the course material.

    (And I *need* to concentrate on the course material. I'm in a class full of
    incredibly smart people with a lot more background in the subject. Pyrex
    helps me keep up with them. So does only taking one class and investing
    ridiculous amounts of time on the homework.)

    Thanks again for helping me avoid C.

    Now to write some "thinning" code...

    --kyler
     
    Kyler Laird, Feb 4, 2004
    #10
  11. Kyler Laird wrote:
    > cdef public char* string(x):
    > s = str(x)
    > return(s)


    Whenever you allow Pyrex to coerce a Python string into a
    char *, you need to be careful that the Python string will
    remain alive for as long as the C pointer is needed.

    In this case, str(x) is probably returning a string with
    no other references, so it's deallocated as soon as the
    function finishes, and a dangling pointer is then returned.

    You'll need to make a copy of the string's contents
    somehow, e.g.

    cdef extern from "string.h":
    char *strcpy(char *)

    cdef public char* string(x):
    s = str(x)
    return strcpy(s)

    and remember to free() it when you're finished with
    it.

    (Warning: Don't try to write it as

    return strcpy(str(x)) # WRONG

    or you'll have the same problem -- the string will
    be deallocated before strcpy() gets a chance to
    copy it!)

    Some other comments:

    * In the C code, it's better to declare pointers to
    Python objects as PyObject * rather than void *, e.g.

    PyObject *im, *im_size;

    since this will give you better type checking from the
    C compiler.

    * Your C code as written leaks Python objects, since it
    never decrefs the objects returned by image_file_open
    and image_size.It also doesn't do anything about checking
    for and handling Python exceptions.

    It's much better if you can do all creation and manipulation
    of Python objects in Pyrex, since it takes care of
    refcounting and error checking automatically (although
    read the doc section on Error Return Values to make sure
    exceptions are propagated from functions that don't
    return Python objects).

    You may want to put something in your main function to
    catch exceptions and print a traceback, since that
    won't happen automatically with no Python interpreter
    at the top level.

    --
    Greg Ewing, Computer Science Dept,
    University of Canterbury,
    Christchurch, New Zealand
    http://www.cosc.canterbury.ac.nz/~greg
     
    Greg Ewing (using news.cis.dfn.de), Feb 12, 2004
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andreas Lagemann
    Replies:
    8
    Views:
    487
    Mike Wahler
    Jan 10, 2005
  2. tiwy
    Replies:
    0
    Views:
    446
  3. =?iso-8859-1?Q?Fran=E7ois?= Pinard

    Re: Pyrex without Python (was Re: calling Pyrex results from C)

    =?iso-8859-1?Q?Fran=E7ois?= Pinard, Jan 21, 2004, in forum: Python
    Replies:
    3
    Views:
    314
    A.M. Kuchling
    Jan 21, 2004
  4. Chris Lambacher
    Replies:
    0
    Views:
    671
    Chris Lambacher
    Jun 8, 2005
  5. Greg Ewing
    Replies:
    2
    Views:
    346
    Dieter Maurer
    Jun 29, 2006
Loading...

Share This Page