checking a thread has started

[Deepak Sarda]

Hello everyone.

I have run into something which I believe is a bug or a shortcoming of
the threading.Thread module.

My program spawns 15 threads. For this I've creating a new class with
threading.Thread as the base class. Then I create objects of this
class and call their start() methods in a loop.

The program works fine when run locally in a shell. The problem starts
when it is served through apache as a cgi program. I just can't get
all the threads running. The constructor is properly called for each
object but for some of them - the run() method never gets called. I
can get around this by introducing a delay in between calls to the
start() methods of the fifteen objects (using time.sleep(2)).

I believe this bottleneck is due to Apache server - it just can't
spawn so many threads so quickly. It just spawns seven threads and
that's it.

My question is: how do I check whether a call to the start() method
was successful? If there is a mechanism by which I can get this
information - I can try restarting/recreating the thread.

I am running this on Python 2.2 and Apache 1.3.27

Regards,

Deepak

 

[Roger Binns]

I believe this bottleneck is due to Apache server - it just can't
spawn so many threads so quickly. It just spawns seven threads and
that's it.

Something else to watch out for which bites occassionally is that
there is an import lock, and only one thread at a time can hold it.
If a thread in Apache is importing your code, then any of your threads
that try to do an import will just block. Your main thread will never
finish and the Apache framework will never finish importing you,
so the lock is never released.

Roger

 

[Michael Fuhr]

Something else to watch out for which bites occassionally is that
there is an import lock, and only one thread at a time can hold it.
If a thread in Apache is importing your code, then any of your threads
that try to do an import will just block. Your main thread will never
finish and the Apache framework will never finish importing you,
so the lock is never released.

What platform is Apache running on? Is the script running under
mod_python or some other framework? The original post mentioned CGI,
which traditionally means a separate process, at least on Unix-like
systems. If the script is running in a separate process from the web
server, then Apache wouldn't be responsible for creating the threads
and it shouldn't be holding any import locks.

 

[Deepak Sarda]

What platform is Apache running on? Is the script running under
mod_python or some other framework? The original post mentioned CGI,
which traditionally means a separate process, at least on Unix-like
systems. If the script is running in a separate process from the web
server, then Apache wouldn't be responsible for creating the threads
and it shouldn't be holding any import locks.

The server is on Redhat with a custom 2.4.20 kernel. The script is not
running through mod_python - just a regular #!/usr/bin/python2.2

I see what you mean and it's true that the script actually runs as a
local user. But there is certainly a relation between the server and
the threads because in httpd.conf - there's a directive which says:

StartServers 8

which means there are 8 idle server instances running at any given
time. Now I assume my main script call take up one instance. Then
when new threads are requested - each thread takes up an instance so
we have 7 new threads. Which is exactly what happens!

The program _always_ successfully creates and completes execution of
exactly seven new threads leaving the remaining eight requested
threads in no man's land!

So there must be some relation between apache and the script. Also, as
I said earlier, when run through a shell as the same user - the script
works perfectly fine.

Deepak

 

[Deepak Sarda]

Something else to watch out for which bites occassionally is that
there is an import lock, and only one thread at a time can hold it.
If a thread in Apache is importing your code, then any of your threads
that try to do an import will just block. Your main thread will never
finish and the Apache framework will never finish importing you,
so the lock is never released.

I am sorry but I didn't understand anything you just said I'll
just explain my code organization..

main.py: at the top - import os, sys, cgi; from core.py import *;
core.py: at the top - import os, sys, threading, time, ...
There are no other import statements anywhere else.

the worker class is defined in core.py and the function which creates
the new threads is is also in core.py.

So with this arrangement, would I be running into import lock
problems? Why wouldn't this problem arise when run directly and not
through apache?

Deepak

 

[Michael Fuhr]

The server is on Redhat with a custom 2.4.20 kernel. The script is not
running through mod_python - just a regular #!/usr/bin/python2.2

The shebang line doesn't control how the web server starts the
script, so that doesn't tell us anything. Does the web server have
mod_python installed at all? If so then it's possible that all
files ending in .py or all files in a certain directory are run
under mod_python.

You can examine os.environ['GATEWAY_INTERFACE'] to see how your
script is run: the value should be "CGI/1.1" if you're running
via CGI and "Python-CGI/1.1" if you're under mod_python. This
environment variable probably won't be set if you run the script
from the command line, so use has_key() or trap KeyError to avoid
exceptions.

It may or may not matter whether you're running under mod_python,
but let's find out for sure anyway. Then we can better understand
the extent of relationship between your script and Apache.

I see what you mean and it's true that the script actually runs as a
local user.

What do you mean "runs as a local user"? CGI programs usually run
as whatever user the web server runs as, although it's possible to
have them run as the script's owner (e.g., by setting up suEXEC).

But there is certainly a relation between the server and
the threads because in httpd.conf - there's a directive which says:

StartServers 8

which means there are 8 idle server instances running at any given
time.

That's not what StartServers means according to the Apache
documentation:

http://httpd.apache.org/docs/mod/core.html#startservers

"The StartServers directive sets the number of child server processes
created on startup. As the number of processes is dynamically
controlled depending on the load, there is usually little reason
to adjust this parameter."

Now I assume my main script call take up one instance. Then
when new threads are requested - each thread takes up an instance so
we have 7 new threads. Which is exactly what happens!

That doesn't make sense given the description of StartServers.
Besides, as I recall, Apache 1.3.27, which you said you were using,
doesn't even use threads -- each instance of the web server is run
in a separate process. Have you tested your hypothesis by altering
the value of StartServers and checking whether your script behaves
differently?

StartServers is 5 on one of my machines, yet I just ran a few
tests and successfully created 1000 threads under both CGI
and mod_python.

The program _always_ successfully creates and completes execution of
exactly seven new threads leaving the remaining eight requested
threads in no man's land!

How are you determining which threads are starting and which aren't?

So there must be some relation between apache and the script. Also, as
I said earlier, when run through a shell as the same user - the script
works perfectly fine.

Something's going on but we lack sufficent evidence to say what.
We haven't seen your code yet -- could you post the smallest possible
script that demonstrates the problem?

 

[Francesco Bochicchio]

My question is: how do I check whether a call to the start() method was
successful? If there is a mechanism by which I can get this information -
I can try restarting/recreating the thread.

I'm not aware of any method in the standard library to do this. But you
could try using events for this purpose, doing soomething like this:

class MyThread( threading.Thread ):
def __init__(self, id ):
threading.Thread.__init__(self)
self.idx = id
self.actually_started = threading.Event()

def run(self):
self.actually_started.set() # signals that the thread is started
# thread processing goes here
print time.time(), "Thread %s started" % self.idx
time.sleep(3.0)
print time.time(), "Thread %d completed" % self.idx
#
# you could try this to serialize the thread creation
# without adding the sleep. It should be faster.
#
def start_threads_serially( num_threads, max_wait = 2.0):
res = []
for i in range(num_threads):
t = MyThread(i)
t.start()
# wait that the thread is started before
# starting the next one
t.actually_started.wait( max_wait )
res.append(t)
return res

# selftest
if __name__ == '__main__':
threadlist = start_threads_serially(100)
for t in threadlist:
t.join()


if __name__ == '__main__':
threadlist = start_threads_serially(10)
for t in threadlist:
t.join()

 

[Roger Binns]

[Please do not email and reply via news as well.]

So with this arrangement, would I be running into import lock
problems?

Unlikely. Google for 'python "import lock"' if you would like
to learn more.

Roger

 

[Deepak Sarda]

Hi Michael. Thanks for the detailed response..

You can examine os.environ['GATEWAY_INTERFACE'] to see how your
script is run: the value should be "CGI/1.1" if you're running
via CGI and "Python-CGI/1.1" if you're under mod_python. This
environment variable probably won't be set if you run the script
from the command line, so use has_key() or trap KeyError to avoid
exceptions.

It give 'CGI/1.1' which means no mod_python

What do you mean "runs as a local user"? CGI programs usually run
as whatever user the web server runs as, although it's possible to
have them run as the script's owner (e.g., by setting up suEXEC).

Yes, it's using the suEXEC mechanism. The script is in my home folder
so it get's executed as my uid/gid.

That doesn't make sense given the description of StartServers.
Besides, as I recall, Apache 1.3.27, which you said you were using,
doesn't even use threads -- each instance of the web server is run
in a separate process. Have you tested your hypothesis by altering
the value of StartServers and checking whether your script behaves
differently?

I couldn't check this yet as the sys admin is not available. but
something related is there in the mail below.

How are you determining which threads are starting and which aren't?

Just seeing if the run() is called - with a print.

Something's going on but we lack sufficent evidence to say what.
We haven't seen your code yet -- could you post the smallest possible
script that demonstrates the problem?

The following script always works for threads less than 8. Increase it
beyond that and it almost always fails. I say almost always because
sometimes it manages to run. But it fails almost 98% of the time
(unscientific observation!). The key thing is the os.system() call.
Without that, everything works fine. Also - if I start the threads
just after creating them - then the success rate become 98% (again -
guess).

#!/usr/bin/python2.2
import sys, os, threading

class Worker(threading.Thread):
def __init__(self, index):
self._index = index
threading.Thread.__init__(self)
print "Made object", self._index
sys.stdout.flush()

def run(self):
print "Ran thread with index", self._index
sys.stdout.flush()
os.system("date")
sys.stdout.flush()
Worker.count += 1

if __name__ == '__main__':
sys.stderr = sys.stdout
print "Content-type: text/plain\n"
sys.stdout.flush()

Worker.count = 0

tlist = []
for i in range(10):
t = Worker(i)
#t.start()
tlist.append(t)

for t in tlist:
t.start()

for t in tlist:
t.join()

print "Count:", Worker.count


I'll experiment with the other suggestions offered and get back soon.

 

[Michael Fuhr]

The following script always works for threads less than 8. Increase it
beyond that and it almost always fails. I say almost always because
sometimes it manages to run. But it fails almost 98% of the time
(unscientific observation!). The key thing is the os.system() call.
Without that, everything works fine. Also - if I start the threads
just after creating them - then the success rate become 98% (again -
guess).

I wonder if you're hitting a resource limit on number of processes;
maybe your CGI environment has lower limits than your shell
environment. Omitting os.system() would reduce the number of
processes you're running, and starting the threads immediately after
creating them would mean that some os.system() calls might finish
before others started, again reducing the number of concurrent
processes. That's one possible explanation that fits the facts,
so let's check it out:

What do you get when you run the following script from the shell
and then as a CGI program?

#!/bin/sh
echo "Content-Type: text/plain"
echo ""
ulimit -a

 

[Deepak Sarda]

What do you get when you run the following script from the shell
and then as a CGI program?

#!/bin/sh
echo "Content-Type: text/plain"
echo ""
ulimit -a

This is the output as a cgi program:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 262144
cpu time (seconds, -t) unlimited
max user processes (-u) 7168
virtual memory (kbytes, -v) unlimited


The only difference is that when run from the shell, the stack size is
also unlimited

Deepak

 

[Deepak Sarda]

I'm not aware of any method in the standard library to do this. But you
could try using events for this purpose, doing soomething like this:

I tried what you suggested and it works... almost!

This is my thread related code block:

while 1:
t = CreateT( some params)
t.start()
t.actually_started.wait(4.0)
if t.actually_started.isSet():
threadList.append(t)
break

And i loop over the params using this block. Now the whole program
runs and prints the expected output - reaches the last line of code.
But it just doesn't exit! I've tried putting a sys.exit() as well as a
raise SystemExit as the last line - the program still doesn't finish.

Out of the 15 threads, two threads had to be created twice before they
were properly started. I suspect it's due to these dangling threads
that the python interpretor can't exit cleanly. I may be totally wrong
there but that's my guess.


Deepak

 

[Michael Fuhr]

max user processes (-u) 7168

That refutes the hypothesis that you've hit a process limit; it
must be something else.

 

[Deepak Sarda]

That refutes the hypothesis that you've hit a process limit; it
must be something else.

I didn't say I've reached process limit. What I suspect is happening
is that it's taking more time than expected for the thread to be
allocated and as a result, I am getting a defunct thread.