multiprocess (and paramiko)

[mennis]

I was able to work around this by using a completely different design but Istill don''t understand why this doesn't work. It appears that the process that launches the process doesn't get access to updated object attributes.. When I set and check them in the object itself it behaves as expected. When I check them from outside the object instance I get the initial valuesonly. Could someone explain what I'm missing?

Here I have a simple multiprocessing class that when initializes takes a connected SSHClient instance and a command to run on the associated host in anew channel.

import multiprocessing

from time import time
from Crypto import Random
import paramiko


class Nonblock(multiprocessing.Process):

def __init__(self, connection, cmd):
Random.atfork()
multiprocessing.Process.__init__(self)

self.transport = connection.get_transport()
if self.transport is None:
raise ConnectionError("connection.get_transport() returned None")
self.channel = self.transport.open_session()

self.command = cmd
self.done = False
self.stdin = None
self.stdout = None
self.stderr = None
self.status = None
self.message = str()
self.time = float()

def _read(self, channelobj):
"""read until EOF"""
buf = channelobj.readline()
output = str(buf)
while buf:
buf = channelobj.readline()
output += buf
return output

def run(self):

start = time()
stdin, stdout, stderr = self.channel.exec_command(command=self.command)

self.stderr = self._read(stderr)
self.status = stdout.channel.recv_exit_status()

if self.status != 0:
self.status = False
self.message = self.stderr
else:
self.status = True
self.message = self._read(stdout)

self.time = time() - start
stdin.close()

self.done = True


I expect to use it in the following manner:

from simplelib import Nonblock
from time import sleep
from paramiko import SSHClient, AutoAddPolicy


if __name__== "__main__":
connection = SSHClient()
connection.set_missing_host_key_policy(AutoAddPolicy())

username = "uname"
hostname = "hostname"
password = "password"

connection.connect(hostname, 22, username, password)
print connection.exec_command("sleep 1; echo test 0")[1].read()

n = Nonblock(connection,"sleep 20; echo test 2")
n.start()

print connection.exec_command("sleep 1; echo test 1")[1].read()
while not n.done:
sleep(1)
print n.message
print "done"

 

[Chris Angelico]

I was able to work around this by using a completely different design butI still don''t understand why this doesn't work. It appears that the process that launches the process doesn't get access to updated object attributes. When I set and check them in the object itself it behaves as expected.When I check them from outside the object instance I get the initial values only. Could someone explain what I'm missing?

When you fork into two processes, the child gets a copy of the
parent's state, but after that, changes happen completely
independently. You need to use actual multiprocessing features like
Queue and such to pass information from one process to another.

ChrisA

 

[Roy Smith]

Here I have a simple multiprocessing class that when initializes takes a
connected SSHClient instance and a command to run on the associated host in a
new channel.

ChrisA has already answered your question, but to answer the question
you didn't ask, you probably want to ditch working directly with
paramiko and take a look at fabric (http://www.fabfile.org/). It layers
a really nice interface on top of paramiko. Instead of gobs of
low-level paramiko code, you just do something like:

from fabric.api import env, run
env.host_string = "my-remote-hostname.com"
output = run("my command")

and you're done.

 

[mennis]

I'm familiar with and have learned much from fabric. Its execution model don't work for this specific interface I'm working on. I use fabric for other things though and it's great.

Ian