multithreading in python

[samaneh.yahyapour]

hi
my program work by 4 thread but when i use more thread it terminates

i use opencv in my image_process.so

my code is :


#!/usr/bin/python
import sys
import os
import io
import time
import copy
import threading
import ctypes

class MyClass():

def __init__(self):
i = 0
while i<10:
thread1 = threading.Thread(target=self.item_thread)
thread1.start()
i = i+1
time.sleep(0.01)

def item_thread(self):
imageAnalyzer=ctypes.CDLL("../so/image_process.so")
imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml", "/opt/amniran/etc/porn.xml")
for filename in os.listdir("../script/images/"):
if filename[-4:] == ".jpg" or filename[-4:] == ".png" or filename[-4:] == ".gif" or filename[-5:] == ".jpeg" :

path = "../script/images/%s"%filename

fo = file(path, "r")
content = fo.read()
score = imageAnalyzer.score_image(content, len(content))
print "%d : %s " %(score, path)
print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"






x = MyClass()



how can i solve this problem

 

[Steven D'Aprano]

hi
my program work by 4 thread but when i use more thread it terminates

Is that a problem? Isn't it supposed to terminate, when it has finished?

If it raises an exception, or crashes, you should tell us.

i use opencv in my image_process.so

my code is :


#!/usr/bin/python
import sys
import os
import io
import time
import copy
import threading
import ctypes

class MyClass():

def __init__(self):
i = 0
while i<10:
thread1 = threading.Thread(target=self.item_thread)
thread1.start()
i = i+1
time.sleep(0.01)

This is better written as:

def __init__(self):
self.threads = []
for i in range(10):
thread = threading.Thread(target=self.item_thread)
thread.start()
self.threads.append(thread)
time.sleep(0.01) # not sure this helps for anything


I think it will also help if you keep references to the threads. That
will stop them from being garbage collected unexpectedly, and you can
check their status before exiting the main thread.

def item_thread(self):
imageAnalyzer=ctypes.CDLL("../so/image_process.so")
imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml",
"/opt/amniran/etc/porn.xml") for filename in
os.listdir("../script/images/"):
if filename[-4:] == ".jpg" or filename[-4:] == ".png" or
filename[-4:] == ".gif" or filename[-5:] == ".jpeg" :

path = "../script/images/%s"%filename

fo = file(path, "r")
content = fo.read()
score = imageAnalyzer.score_image(content, len(content))
print "%d : %s " %(score, path)
print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"


x = MyClass()

I suspect that when the main thread exits, and your other threads are
still running, you may be in trouble. But I'm not a threading expert, so
I could be wrong. However, I would put something like this at the end:

for thread in x.threads:
x.join()


that way the main thread cannot finish until each of the subthreads are.

 

[Dave Angel]

hi
my program work by 4 thread but when i use more thread it terminates

I simplified your code so anybody could run it, and tested it inside
Komodo IDE, on Python 2.7

#!/usr/bin/env python


import sys
import os
import time
import threading

class MyClass():

def __init__(self):
i = 0
while i<10:
work = WorkClass(i)
thread1 = threading.Thread(target=work.item_thread)
thread1.start()
i = i+1
time.sleep(0.01)

class WorkClass():
def __init__(self, parm):
self.parm = str(parm)
def item_thread(self):
print "beginning thread", self.parm
for filename in os.listdir("."):
data = "thread " + self.parm + " filename " + filename + "\n"
print data
time.sleep(0.5)
print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD " + self.parm

x = MyClass()
print "Finishing main thread"

When the

 

[Dave Angel]

hi
my program work by 4 thread but when i use more thread it terminates
<snip>
how can i solve this problem

I simplified the code so I could actually run it, and tested it in
Python 2.7, both under Komodo IDE and in the terminal.

The code:

#!/usr/bin/env python


import sys
import os
import time
import threading

class MyClass():

def __init__(self):
self.threads = []
i = 0
while i<10:
work = WorkClass(i)
thread1 = threading.Thread(target=work.item_thread)
self.threads.append(thread1)
thread1.start()
i = i+1
time.sleep(0.01)
for thread in self.threads: #wait for all the threads to end
thread.join()

class WorkClass():
def __init__(self, parm):
self.parm = str(parm)
def item_thread(self):
print "beginning thread", self.parm
for filename in os.listdir("."):
data = "thread " + self.parm + " filename " + filename + "\n"
print data
time.sleep(0.5)
print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD " + self.parm

x = MyClass()
print "All done with main thread"

The original code couldn't tell us what threads were running, so the
only clue you got was how many times it printed "ENDDDD" So I arranged
that each thread had a unique "parm" value. Typically you must do
something like this so that all the threads aren't doing exactly the
same work.

Another thing i did was to "atomicize" the print statements. As it
originally was, partial lines from different threads could be
intermixed in the output.

The IDE showed the error message:

"""
ERROR: dbgp.client:
The main thread of this application is exiting while there are still threads
alive. When the main thread exits, it is system defined whether the other
threads survive.

See Caveats at http://docs.python.org/lib/module-thread.html
"""

which tells you what's wrong. You need to do a join on the threads
before exiting.

it's traditional (and better) to derive your own class from
threading.Thread, and that's where you can store any additional
attributes that each thread will need. I demonstrated something I
figured was simpler, by making the item_thread() method part of a
separate class.

 

[Terry Reedy]

On 8/13/2013 4:06 AM, (e-mail address removed) wrote:

Aside from the other comments...

def item_thread(self):
imageAnalyzer=ctypes.CDLL("../so/image_process.so")
imageAnalyzer.aref_img_score_init("/opt/amniran/etc/face.xml", "/opt/amniran/etc/porn.xml")
for filename in os.listdir("../script/images/"):
if filename[-4:] == ".jpg" or filename[-4:] == ".png" or filename[-4:] == ".gif" or filename[-5:] == ".jpeg" :

path = "../script/images/%s"%filename

fo = file(path, "r")

Use 'open' instead of the deprecated 'file' (removed in 3.x) and use it
with a 'with' statement (this is now standard). This closes the file at
the end of the block. Not doing so can cause problems on other
implementations.

with open(path, 'rb') as fo:

content = fo.read()

score = imageAnalyzer.score_image(content, len(content))
print "%d : %s " %(score, path)
print "ENDDDDDDDDDDDDDDDDDDDDDDDDDDDD"

Do you know how to use queue.Queue to spread work to multiple worker
threads so each processes different files (and to collect results from
multiple threads).? (If not, read doc.)