Open Multiples Files at same time with multiprocessing - How"declare" dynamically the var?

[macm]

Hi Folks

My approach to open multiples files at same time is:

def openFiles(self,file,q):
fp = open(file, 'rb')
fp.seek(0)
fcontent = fp.read()
fp.close()
q.put(fcontent)
return

def testOpen(self):
L =
['file1.txt','file2.txt','file3.txt','file4.txt','file5.txt']
d1 = []
for x in L:
z=L.index(x)
q = Queue()
m = Process(target=self.openFiles, args=(x,q,))
m.start()
d1.append(q.get()) # <= This get is locking ? It is mean:
"wait m.start(), like m.join()??"

print list(d1)
return

Is the best way? Is q.get() locking the loop?

I feel that q.get() is locking so I would like to "declare"
dynamically the var q{z} in python? is it possible? How?

def testOpen(self):
L =
['file1.txt','file2.txt','file3.txt','file4.txt','file5.txt']
d1 = []
for x in L:
z=L.index(x)
q{z} = Queue()
m{z} = Process(target=self.openFiles, args=(x,q{z},))
m{z}.start()

for x in L:
z=L.index(x)
d1.append(q{z}.get()) # <= So now I am sure that q{z}.get() isn't
lock

print list(d1)
return

I tried use list but didn't work look below one shot.

Best Regards

macm

I tried :

def testOpen(self):
L = ['file1.txt','file2.txt',
'file3.txt','file4.txt',
'file5.txt']
d1 = []
q = []
m = []
for x in L:
z=L.index(x)
q.insert(z, Queue())
m.insert(z,Process(target=self.openFiles, args=(x,q[z])))
m[z].start()

# Now I am sure. Isnt lock
for x in L:
z=L.index(x)
d1.append(q[z].get())

print list(d1)
return

 

[Ian Kelly]

def openFiles(self,file,q):
fp = open(file, 'rb')
fp.seek(0)

The seek is unnecessary; the file will already be at position 0 after it
is opened.

def testOpen(self):
L =
['file1.txt','file2.txt','file3.txt','file4.txt','file5.txt']
d1 = []
for x in L:
z=L.index(x)

"for z, x in enumerate(L):"

q = Queue()
m = Process(target=self.openFiles, args=(x,q,))
m.start()
d1.append(q.get()) #<= This get is locking ? It is mean:
"wait m.start(), like m.join()??"

It can't get an item from the queue until an item has been put in the
queue to get, so it waits for the process m to put something there. It
does not do an implicit m.join() however.

I tried use list but didn't work look below one shot.

"It didn't work" is not very useful for helping you debug. Be specific.
What were you expecting, and what did you get instead? If there was a
traceback, copy and paste it.

Cheers,
Ian