moving Connection/PipeConnection between processes

Discussion in 'Python' started by Randall Smith, Jun 13, 2009.

  1. I've got a situation in which I'd like to hand one end of a pipe to
    another process. First, in case you ask why, a spawner process is
    created early before many modules are imported. That spawner process is
    responsible for creating new processes and giving a proxy to the parent

    (h1-1 <-- Pipe() --> h1-2)
    |------------------------> child1
    |(s1 < Pipe() > s2)
    parent -> spawner -> |--> child2
    |(h2-1) <-- Pipe() --> (h2-2)
    |------------------------> child3
    (h3-1 <-- Pipe() --> h3-2)

    When I try to pass Connection h1-1 to the parent using Connection s1, I
    get an error:

    TypeError: Required argument 'handle' (pos 1) not found

    This error is from unpickling the Connection h1-1. You can duplicate
    the error like this:

    pickled_connection = pickle.dumps(h1-1, 2)
    pickle.loads(pickled_connection) # raises the same error

    Looking at the pickle docs, I wonder if this could be resolved by adding
    a __getnewargs__ method to _multiprocessing.Connection. But even if
    that would work I couldn't do it now since it's an extension module.
    I've thought about trying to recreate the Connection. Looks like it
    should be possible with Connection.fileno(). The Unix portion looks
    easy, but the win32 portion does not.

    So if it's possible, what's the best way to pass a Connection to another

    Randall Smith, Jun 13, 2009
    1. Advertisements

  2. Pickle has nothing to do with the problem since it lay much deeper: in
    the OS.

    From kernel point of view, every process has it's own "descriptor
    table" and the integer id of the descriptor is all the process gets, so
    when you say "os.pipe()" kernel actually gives you a number which is
    completely meaningless for any other process - it either doesn't exists
    in it's descriptor table or points to something else.

    So, what you actually need is to tell the kernel to duplicate
    underlying object in another process' table (with it's own numbering),
    which is usually done via special flag for sendmsg(2) in C, so you
    should probably look out for py implementation of this call, which I
    haven't stumbled upon, but, admittely, never looked for.

    Mike Kazantsev //

    Version: GnuPG v2.0.11 (GNU/Linux)

    -----END PGP SIGNATURE-----
    Mike Kazantsev, Jun 13, 2009
    1. Advertisements

  3. Randall Smith

    jenifer adam Guest

    Check its a good one.
    jenifer adam, Jun 13, 2009
  4. As I was referring to earlier, Unix is easy.

    fd = mycon.fileno()
    new_con = _multiprocessing.Connection(os.dup(fd))

    But Windows?

    The implementation of the pipe creation is on line 167 of
    I don't know where to start.

    Randall Smith, Jun 13, 2009
  5. Now that I've done some homework, everything you said is clear.

    sendmsg is a missing feature of the socket module.

    And this implements it:
    Randall Smith, Jun 15, 2009
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.