basic thread question

[Derek Martin]

I have some simple threaded code... If I run this
with an arg of 1 (start one thread), it pegs one cpu, as I would
expect. If I run it with an arg of 2 (start 2 threads), it uses both
CPUs, but utilization of both is less than 50%. Can anyone explain
why?

I do not pretend it's impeccable code, and I'm not looking for a
critiqe of the code per se, excepting the case where what I've written
is actually *wrong*. I hacked this together in a couple of minutes,
with the intent of pegging my CPUs. Performance with two threads is
actually *worse* than with one, which is highly unintuitive. I can
accomplish my goal very easily with bash, but I still want to
understand what's going on here...

The OS is Linux 2.6.24, on a Ubuntu base. Here's the code:

Thanks

-=-=-=-=-

#!/usr/bin/python

import thread, sys, time

def busy(thread):
x=0
while True:
x+=1

if __name__ == '__main__':
try:
cpus = int(sys.argv[1])
except ValueError:
cpus = 1
print "cpus = %d, argv[1] = %s\n" % (cpus, sys.argv[1])
i=0
thread_list = []
while i < cpus:
x = thread.start_new_thread(busy, (i,))
thread_list.append(x)
i+=1
while True:
pass

--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFKiwqndjdlQoHP510RAtMjAJwMI77kgLeJ6BwPCzpa0X0S+ufNDACfXI5j
JwDqKbMirN5bknUDvZ8yWYQ=
=KbeP
-----END PGP SIGNATURE-----

 

[birdsong]

I have some simple threaded code...  If I run this
with an arg of 1 (start one thread), it pegs one cpu, as I would
expect.  If I run it with an arg of 2 (start 2 threads), it uses both
CPUs, but utilization of both is less than 50%.  Can anyone explain
why?  

I do not pretend it's impeccable code, and I'm not looking for a
critiqe of the code per se, excepting the case where what I've written
is actually *wrong*. I hacked this together in a couple of minutes,
with the intent of pegging my CPUs.  Performance with two threads is
actually *worse* than with one, which is highly unintuitive.  I can
accomplish my goal very easily with bash, but I still want to
understand what's going on here...

The OS is Linux 2.6.24, on a Ubuntu base.  Here's the code:

Thanks

-=-=-=-=-

#!/usr/bin/python

import thread, sys, time

def busy(thread):
    x=0
    while True:
        x+=1

if __name__ == '__main__':
    try:
        cpus = int(sys.argv[1])
    except ValueError:
        cpus = 1
    print "cpus = %d, argv[1] = %s\n" % (cpus, sys.argv[1])
    i=0
    thread_list = []
    while i < cpus:
        x = thread.start_new_thread(busy, (i,))
        thread_list.append(x)
        i+=1
    while True:
        pass

--
Derek D. Martinhttp://www.pizzashack.org/
GPG Key ID: 0x81CFE75D

 application_pgp-signature_part
< 1KViewDownload

watch this and all your findings will be explained: http://blip.tv/file/2232410

this talk marked a pivotal moment in my understanding of python
threads and signal handling in threaded programs.

 

[sturlamolden]

I have some simple threaded code...  If I run this
with an arg of 1 (start one thread), it pegs one cpu, as I would
expect.  If I run it with an arg of 2 (start 2 threads), it uses both
CPUs, but utilization of both is less than 50%.  Can anyone explain
why?  

Access to the Python interpreter is serialized by the global
interpreter lock (GIL). You created two threads the OS could schedule,
but they competed for access to the Python interpreter. If you want to
utilize more than one CPU, you have to release the GIL or use multiple
processes instead (os.fork since you are using Linux).

This is how the GIL can be released:

* Many functions in Python's standard library, particularly all
blocking i/o functions, release the GIL. This covers the by far most
common use of threads.

* In C or C++ extensions, use the macros Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS.

* With ctypes, functions called from a cdll release the GIL, whereas
functions called from a pydll do not.

* In f2py, declaring a Fortran function threadsafe in a .pyf file or
cf2py comment releases the GIL.

* In Cython or Pyrex extensions, use a "with nogil:" block to execute
code without holding the GIL.


Sturla Molden