Creating a python extension that works with multiprocessing.Queue

Discussion in 'Python' started by Travis Miller, Mar 15, 2009.

  1. I am very new to the python C API, and have written a simple type
    called SU2 that has 4 members that are all doubles. Everything seems
    to work fine (can import my module and instantiate the new type and
    act on it with various methods I have defined), except for when I
    attempt to use my new type with multiprocessing.Queue (by the way I am
    working with python 2.6).

    So when I declare my object instance and call the put() method on the
    Queue instance, the object I get out with the get() method of the same
    Queue instance is not the same object. The four members of the
    object are all coming through as all zeros. Below is the snippet of
    code.

    from multiprocessing import Queue, Process
    from gauge import SU2 # this is my new module and type

    def func(q):
    s = SU2(1.0, 2.0, 3.0, 4.0)
    q.put(s)

    q = Queue()
    p = Process(target=func, args=(q,))
    p.start()
    s = q.get()
    p.join()


    The problem is that the s I get in the get() method has its members
    set to 0.0 rather than what I set in func(). So the members are not
    being put in the Queue correctly. I have defined the SU2 type using
    the python C API and so I suspect I have neglected to define a method
    that is needed by Queue to do its stuff. What am I missing?

    Travis Miller
     
    Travis Miller, Mar 15, 2009
    #1
    1. Advertising

  2. En Sun, 15 Mar 2009 01:51:35 -0200, Travis Miller <>
    escribió:

    > I am very new to the python C API, and have written a simple type
    > called SU2 that has 4 members that are all doubles. Everything seems
    > to work fine (can import my module and instantiate the new type and
    > act on it with various methods I have defined), except for when I
    > attempt to use my new type with multiprocessing.Queue (by the way I am
    > working with python 2.6).
    >
    > So when I declare my object instance and call the put() method on the
    > Queue instance, the object I get out with the get() method of the same
    > Queue instance is not the same object. The four members of the
    > object are all coming through as all zeros. Below is the snippet of
    > code.


    Are you on Windows? multiprocessing uses pickles to transfer objects
    between processes. See if you can dump and load those kind of objects. If
    not, you may need to implement __getstate__/__setstate__ or __reduce__
    See http://docs.python.org/library/pickle.html


    --
    Gabriel Genellina
     
    Gabriel Genellina, Mar 18, 2009
    #2
    1. Advertising

  3. I'm on linux actually. I found that so long as I implement
    __reduce__, then pickling works, and my extension works with Queue
    objects correctly. So far the C Api is really cool. I can do all the
    math stuff where I need the speed, and still be able to use python
    which is way more expressive.

    travis

    On Mar 17, 8:43 pm, "Gabriel Genellina" <>
    wrote:
    > En Sun, 15 Mar 2009 01:51:35 -0200, Travis Miller <>  
    > escribió:
    >
    > > I am very new to the python C API, and have written a simple type
    > > called SU2 that has 4 members that are all doubles.  Everything seems
    > > to work fine (can import my module and instantiate the new type and
    > > act on it with various methods I have defined), except for when I
    > > attempt to use my new type with multiprocessing.Queue (by the way I am
    > > working with python 2.6).

    >
    > > So when I declare my object instance and call the put() method on the
    > > Queue instance, the object I get out with the get() method of the same
    > > Queue instance is not the same object.   The four members of the
    > > object are all coming through as all zeros.  Below is the snippet of
    > > code.

    >
    > Are you on Windows? multiprocessing uses pickles to transfer objects  
    > between processes. See if you can dump and load those kind of objects. If  
    > not, you may need to implement __getstate__/__setstate__ or __reduce__
    > Seehttp://docs.python.org/library/pickle.html
    >
    > --
    > Gabriel Genellina
     
    Travis Miller, Mar 20, 2009
    #3
  4. En Fri, 20 Mar 2009 00:16:29 -0200, Travis Miller <>
    escribió:

    > So far the C Api is really cool. I can do all the
    > math stuff where I need the speed, and still be able to use python
    > which is way more expressive.


    Sure! One gets the best of both worlds that way.

    --
    Gabriel Genellina
     
    Gabriel Genellina, Mar 20, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    721
    Russell Warren
    Jun 27, 2006
  2. redbaron
    Replies:
    1
    Views:
    361
    redbaron
    Oct 15, 2008
  3. redbaron
    Replies:
    3
    Views:
    554
    Paul Rubin
    Oct 21, 2008
  4. Luis Zarrabeitia

    Multiprocessing.Queue - I want to end.

    Luis Zarrabeitia, Apr 30, 2009, in forum: Python
    Replies:
    3
    Views:
    1,247
  5. Kris
    Replies:
    0
    Views:
    532
Loading...

Share This Page