multiprocessing module

[makobu]

I have a function that makes two subprocess.Popen() calls on a file.

I have 8 cores. I need 8 instances of that function running in
parallel at any given time till all the files are worked on.
Can the multiprocessing module do this? If so, whats the best method?

A technical overview of how the multiprocessing module actually works
would also be really helpful.

regards,
mak.

 

[Aahz]

I have a function that makes two subprocess.Popen() calls on a file.

I have 8 cores. I need 8 instances of that function running in
parallel at any given time till all the files are worked on.
Can the multiprocessing module do this? If so, whats the best method?

You don't quite explicitly say so, but it sounds like you have multiple
files. In which case, yes, it should be reasonably straightforward to
use multiprocessing; I haven't used it myself, but what you want must be
whatever is equivalent to Queue.Queue().

A technical overview of how the multiprocessing module actually works
would also be really helpful.

Not having any clue what you already know makes that rather a daunting
prospect, as indicated by the lack of replies. Maybe you could
summarize?

 

[Glazner]

I have a function that makes two subprocess.Popen() calls on a file.

I have 8 cores. I need 8 instances of that function running in
parallel at any given time till all the files are worked on.
Can the multiprocessing module do this? If so, whats the best method?

A technical overview of how the multiprocessing module actually works
would also be really helpful.

regards,
mak.

I guess you need the Map-Reduce pattern.
It appears like multiprocessing.Pool will do the trick.

def doSomething(f):
pass

pool = Pool(8)# 8 processes 1 per core

files = ['pop.txt','looper.txt','foo.bar']
results = pool.map(doSomething,files) #this does all the job

got it?