moving Connection/PipeConnection between processes

R

Randall Smith

I've got a situation in which I'd like to hand one end of a pipe to
another process. First, in case you ask why, a spawner process is
created early before many modules are imported. That spawner process is
responsible for creating new processes and giving a proxy to the parent
process.

(h1-1 <-- Pipe() --> h1-2)
|------------------------> child1
|(s1 < Pipe() > s2)
parent -> spawner -> |--> child2
|(h2-1) <-- Pipe() --> (h2-2)
|----------------------|
|------------------------> child3
(h3-1 <-- Pipe() --> h3-2)

When I try to pass Connection h1-1 to the parent using Connection s1, I
get an error:

TypeError: Required argument 'handle' (pos 1) not found

This error is from unpickling the Connection h1-1. You can duplicate
the error like this:

pickled_connection = pickle.dumps(h1-1, 2)
pickle.loads(pickled_connection) # raises the same error

Looking at the pickle docs, I wonder if this could be resolved by adding
a __getnewargs__ method to _multiprocessing.Connection. But even if
that would work I couldn't do it now since it's an extension module.
I've thought about trying to recreate the Connection. Looks like it
should be possible with Connection.fileno(). The Unix portion looks
easy, but the win32 portion does not.

So if it's possible, what's the best way to pass a Connection to another
process?

Randall
 
M

Mike Kazantsev

I've got a situation in which I'd like to hand one end of a pipe to
another process. First, in case you ask why, a spawner process is
created early before many modules are imported. That spawner process is
responsible for creating new processes and giving a proxy to the parent
process.
....

Looking at the pickle docs, I wonder if this could be resolved by adding
a __getnewargs__ method to _multiprocessing.Connection. But even if
that would work I couldn't do it now since it's an extension module.
I've thought about trying to recreate the Connection. Looks like it
should be possible with Connection.fileno(). The Unix portion looks
easy, but the win32 portion does not.

So if it's possible, what's the best way to pass a Connection to another
process?

Pickle has nothing to do with the problem since it lay much deeper: in
the OS.

From kernel point of view, every process has it's own "descriptor
table" and the integer id of the descriptor is all the process gets, so
when you say "os.pipe()" kernel actually gives you a number which is
completely meaningless for any other process - it either doesn't exists
in it's descriptor table or points to something else.

So, what you actually need is to tell the kernel to duplicate
underlying object in another process' table (with it's own numbering),
which is usually done via special flag for sendmsg(2) in C, so you
should probably look out for py implementation of this call, which I
haven't stumbled upon, but, admittely, never looked for.


--
Mike Kazantsev // fraggod.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)

iEYEARECAAYFAkozX0gACgkQASbOZpzyXnGxcQCgqJiYRSN3o92OVViG0WO3HZyv
h2sAnibZuUIsEpY4BxxDqkkHUhRB8sXF
=DFN5
-----END PGP SIGNATURE-----
 
J

jenifer adam

Pickle has nothing to do with the problem since it lay much deeper: in
the OS.

From kernel point of view, every process has it's own "descriptor
table" and the integer id of the descriptor is all the process gets, so
when you say "os.pipe()" kernel actually gives you a number which is
completely meaningless for any other process - it either doesn't exists
in it's descriptor table or points to something else.

So, what you actually need is to tell the kernel to duplicate
underlying object in another process' table (with it's own numbering),
which is usually done via special flag for sendmsg(2) in C, so you
should probably look out for py implementation of this call, which I
haven't stumbled upon, but, admittely, never looked for.

--
Mike Kazantsev // fraggod.net

 signature.asc
< 1KViewDownload

Check http://www.voipsipsdk.com its a good one.
 
R

Randall Smith

Mike said:
Pickle has nothing to do with the problem since it lay much deeper: in
the OS.

From kernel point of view, every process has it's own "descriptor
table" and the integer id of the descriptor is all the process gets, so
when you say "os.pipe()" kernel actually gives you a number which is
completely meaningless for any other process - it either doesn't exists
in it's descriptor table or points to something else.

So, what you actually need is to tell the kernel to duplicate
underlying object in another process' table (with it's own numbering),
which is usually done via special flag for sendmsg(2) in C, so you
should probably look out for py implementation of this call, which I
haven't stumbled upon, but, admittely, never looked for.

As I was referring to earlier, Unix is easy.

fd = mycon.fileno()
new_con = _multiprocessing.Connection(os.dup(fd))

But Windows?

The implementation of the pipe creation is on line 167 of connection.py.
I don't know where to start.

Randall
 
R

Randall Smith

Now that I've done some homework, everything you said is clear.

Mike said:
Pickle has nothing to do with the problem since it lay much deeper: in
the OS.

From kernel point of view, every process has it's own "descriptor
table" and the integer id of the descriptor is all the process gets, so
when you say "os.pipe()" kernel actually gives you a number which is
completely meaningless for any other process - it either doesn't exists
in it's descriptor table or points to something else.

So, what you actually need is to tell the kernel to duplicate
underlying object in another process' table (with it's own numbering),
which is usually done via special flag for sendmsg(2) in C, so you
should probably look out for py implementation of this call, which I
haven't stumbled upon, but, admittely, never looked for.

sendmsg is a missing feature of the socket module.

http://bugs.python.org/issue1194378

And this implements it:

http://pypi.python.org/pypi/python-eunuchs/0.0.0
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top