Newbie help for using multiprocessing and subprocess packages forcreating child processes


R

Rob Newman

Hi All,

I am new to Python, and have a very specific task to accomplish. I
have a command line shell script that takes two arguments:

create_graphs.sh -v --sta=STANAME

where STANAME is a string 4 characters long.

create_graphs creates a series of graphs using Matlab (among other 3rd
party packages).

Right now I can run this happily by hand, but I have to manually
execute the command for each STANAME. What I want is to have a Python
script that I pass a list of STANAMEs to, and it acts like a daemon
and spawns as many child processes as there are processors on my
server (64), until it goes through all the STANAMES (about 200).

I posted a message on Stack Overflow (ref: http://stackoverflow.com/questions/...sses-on-a-multi-processor-system-use-multipro)
and was recommended to use the multiprocessing and subprocess
packages. In the Stack Overflow answers, it was suggested that I use
the process pool class in multiprocessing. However, the server I have
to use is a Sun Sparc (T5220, Sun OS 5.10) and there is a known issue
with sem_open() (ref: http://bugs.python.org/issue3770), so it appears
I cannot use the process pool class.

So, below is my script (controller.py) that I have attempted to use as
a test, that just calls the 'ls' command on a file I know exists
rather than firing off my shell script (which takes ~ 10 mins to run
per STANAME):

#!/path/to/python

import sys
import os
import json
import multiprocessing
import subprocess

def work(verbose,staname):
print 'function:',staname
print 'parent process:', os.getppid()
print 'process id:', os.getpid()
print "ls /path/to/file/"+staname+"_info.pf"
# cmd will eventually get replaced with the shell script with the
verbose and staname options
cmd = [ "ls /path/to/file/"+staname+"_info.pf" ]
return subprocess.call(cmd, shell=False)

if __name__ == '__main__':

report_sta_list = ['B10A','B11A','BNLO']

# Print out the complete station list for testing
print report_sta_list

# Get the number of processors available
num_processes = multiprocessing.cpu_count()

print 'Number of processes: %s' % (num_processes)

print 'Now trying to assign all the processors'

threads = []

len_stas = len(report_sta_list)

print "+++ Number of stations to process: %s" % (len_stas)

# run until all the threads are done, and there is no data left
while len(threads) < len(report_sta_list):

# if we aren't using all the processors AND there is still data
left to
# compute, then spawn another thread

print "+++ Starting to set off all child processes"

if( len(threads) < num_processes ):

this_sta = report_sta_list.pop()

print "+++ Station is %s" % (this_sta)

p = multiprocessing.Process(target=work,args=['v',this_sta])

p.start()

print p, p.is_alive()

threads.append(p)

else:

for thread in threads:

if not thread.is_alive():

threads.remove(thread)

However, I seem to be running into a whole series of errors:

myhost{rt}62% controller.py
['B10A', 'B11A', 'BNLO']
Number of processes: 64
Now trying to assign all the processors
+++ Number of stations to process: 3
+++ Starting to set off all child processes
+++ Station is BNLO
<Process(Process-1, started)> True
+++ Starting to set off all child processes
+++ Station is B11A
function: BNLO
parent process: 22341
process id: 22354
ls /path/to/file/BNLO_info.pf
<Process(Process-2, started)> True
function: B11A
parent process: 22341
process id: 22355
ls /path/to/file/B11A_info.pf
Process Process-1:
Traceback (most recent call last):
File "/opt/csw/lib/python/multiprocessing/process.py", line 231, in
_bootstrap
self.run()
File "/opt/csw/lib/python/multiprocessing/process.py", line 88, in
run
self._target(*self._args, **self._kwargs)
File "controller.py", line 104, in work
return subprocess.call(cmd, shell=False)
File "/opt/csw/lib/python/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/opt/csw/lib/python/subprocess.py", line 595, in __init__
errread, errwrite)
File "/opt/csw/lib/python/subprocess.py", line 1092, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Process Process-2:
Traceback (most recent call last):
File "/opt/csw/lib/python/multiprocessing/process.py", line 231, in
_bootstrap
self.run()
File "/opt/csw/lib/python/multiprocessing/process.py", line 88, in
run
self._target(*self._args, **self._kwargs)
File "controller.py", line 104, in work
return subprocess.call(cmd, shell=False)
File "/opt/csw/lib/python/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/opt/csw/lib/python/subprocess.py", line 595, in __init__
errread, errwrite)
File "/opt/csw/lib/python/subprocess.py", line 1092, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

The files are there:

mhost{me}11% ls -la /path/to/files/BNLO_info.pf
-rw-rw-r-- 1 me group 391 May 19 22:40 /path/to/files/
BNLO_info.pf
myhost{me}12% ls -la /path/to/file/B11A_info.pf
-rw-rw-r-- 1 me group 391 May 19 22:27 /path/to/files/
B11A_info.pf

I might be doing this completely wrong, but I thought this would be
the way to list the files dynamically. Admittedly this is just a
stepping stone to running the actual shell script I want to run. Can
anyone point me in the right direction or offer any advice for using
these packages?

Thanks in advance for any help or insight.
- Rob
 
Ad

Advertisements


Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top