Multiprocessing.connection magic

[Claudiu Popa]

Hello guys,
While working at a dispatcher using
multiprocessing.connection.Listener module I've stumbled upon some
sort of magic trick that amazed me. How is this possible and
what does multiprocessing library doing in background for this to
work?

Client, Python 2.6


Server, 3.2Help on function copy in module shutil:

copy(src, dst)
Copy data and mode bits ("cp src dst").

The destination may be a directory.

 

[Chris Torek]

Hello guys,
While working at a dispatcher using
multiprocessing.connection.Listener module I've stumbled upon some
sort of magic trick that amazed me. How is this possible and
what does multiprocessing library doing in background for this to
work?

Most of Python's sharing routines (including multiprocessing
"send", in this case) use the pickle routines to package data
for transport between processes.

Thus, you can "see the magic" pretty simply:

Client, Python 2.6

Here I just use pickle.dumps() to return (and print, since we are
in the interpreter) the string representation that client.send()
will send:

Server, 3.2
Help on function copy in module shutil:

[snip]

On this end, the (different) version of python simply unpickles the
byte stream. Starting a new python session (to get rid of any
previous imports):

$ python
...Help on function copy in module shutil:
...

The real magic is in the unpickler, which has figured out how to
access shutil.copy without importing shutil into the global namespace:
Traceback (most recent call last):

but we can expose that magic as well, by feeding pickle.loads()
a "bad" string:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1124, in find_class
__import__(module)
ImportError: No module named NotAModule
Note the rather total lack of security here -- in the receiver, by
doing con.recv(), you are trusting the sender not to send you a
"dangerous" or invalid pickle-data-stream. This is why the documentation
includes the following:

Warning: The Connection.recv() method automatically unpickles
the data it receives, which can be a security risk unless you
can trust the process which sent the message.

Therefore, unless the connection object was produced using Pipe()
you should only use the recv() and send() methods after performing
some sort of authentication. See Authentication keys.

(i.e., do that -- see the associated section on authentication)

 

[Chris Angelico]

The real magic is in the unpickler, which has figured out how to
access shutil.copy without importing shutil into the global namespace:

So from this I gather that it doesn't actually pickle the code, just
the name. Seems a little odd, but that would explain why this didn't
really work:
x.append(len(x))
return len(x)
b'\x80\x03c__main__\nasdf\nq\x00.'

I tried to do the classic - encode something, discard the original,
attempt to decode. Didn't work.

Chris Angelico

 

[Thomas Rachel]

Am 03.06.2011 08:28 schrieb Claudiu Popa:

Hello guys,
While working at a dispatcher using
multiprocessing.connection.Listener module I've stumbled upon some
sort of magic trick that amazed me. How is this possible and
what does multiprocessing library doing in background for this to
work?

As Chris already said, it probably uses pickle. Doing so, you should be
aware that unpickling strings can execute arbitrary code. So be very
careful if you use something like that...


Thomas

 

[Chris Angelico]

Am 03.06.2011 08:28 schrieb Claudiu Popa:

As Chris already said, it probably uses pickle. Doing so, you should be
aware that unpickling strings can execute arbitrary code. So be very careful
if you use something like that...

Nice piece of safe ambiguity there - two people said that, both named Chris!

Just how many Chrises are there on this list? I have a pet theory that
there's a greater-than-usual correlation between geeks and the name
"Chris", and the Python list has provided a number of supporting
instances.

Chris Angelico

 

[Steven D'Aprano]

Just how many Chrises are there on this list? I have a pet theory that
there's a greater-than-usual correlation between geeks and the name
"Chris", and the Python list has provided a number of supporting
instances.

My theory is that geeks (at least in Australia) gravitate towards the
names Matt, or sometimes Ben. So much so that when I'm interviewing a new
coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of
confusion. Mind if we call you Matt?"

 

[Chris Angelico]

My theory is that geeks (at least in Australia) gravitate towards the
names Matt, or sometimes Ben. So much so that when I'm interviewing a new
coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of
confusion. Mind if we call you Matt?"

Interesting. I'll have to keep my eyes open for the Matts and Bens. Fascinating.

Chris Angelico