Creating a python extension that works with multiprocessing.Queue

[Travis Miller]

I am very new to the python C API, and have written a simple type
called SU2 that has 4 members that are all doubles. Everything seems
to work fine (can import my module and instantiate the new type and
act on it with various methods I have defined), except for when I
attempt to use my new type with multiprocessing.Queue (by the way I am
working with python 2.6).

So when I declare my object instance and call the put() method on the
Queue instance, the object I get out with the get() method of the same
Queue instance is not the same object. The four members of the
object are all coming through as all zeros. Below is the snippet of
code.

from multiprocessing import Queue, Process
from gauge import SU2 # this is my new module and type

def func(q):
s = SU2(1.0, 2.0, 3.0, 4.0)
q.put(s)

q = Queue()
p = Process(target=func, args=(q,))
p.start()
s = q.get()
p.join()


The problem is that the s I get in the get() method has its members
set to 0.0 rather than what I set in func(). So the members are not
being put in the Queue correctly. I have defined the SU2 type using
the python C API and so I suspect I have neglected to define a method
that is needed by Queue to do its stuff. What am I missing?

Travis Miller

 

[Gabriel Genellina]

I am very new to the python C API, and have written a simple type
called SU2 that has 4 members that are all doubles. Everything seems
to work fine (can import my module and instantiate the new type and
act on it with various methods I have defined), except for when I
attempt to use my new type with multiprocessing.Queue (by the way I am
working with python 2.6).

So when I declare my object instance and call the put() method on the
Queue instance, the object I get out with the get() method of the same
Queue instance is not the same object. The four members of the
object are all coming through as all zeros. Below is the snippet of
code.

Are you on Windows? multiprocessing uses pickles to transfer objects
between processes. See if you can dump and load those kind of objects. If
not, you may need to implement __getstate__/__setstate__ or __reduce__
See http://docs.python.org/library/pickle.html

 

[Travis Miller]

I'm on linux actually. I found that so long as I implement
__reduce__, then pickling works, and my extension works with Queue
objects correctly. So far the C Api is really cool. I can do all the
math stuff where I need the speed, and still be able to use python
which is way more expressive.

travis

 

[Gabriel Genellina]

So far the C Api is really cool. I can do all the
math stuff where I need the speed, and still be able to use python
which is way more expressive.

Sure! One gets the best of both worlds that way.