thread/queue bug

Discussion in 'Python' started by phil, Dec 10, 2004.

  1. phil

    phil Guest

    And sorry I got ticked, frustrating week

    >And I could help more, being fairly experienced with
    >threading issues and race conditions and such, but
    >as I tried to indicate in the first place, you've
    >provided next to no useful (IMHO) information to
    >let anyone help you more than this


    This is about 5% of the code.
    Uses no locks.
    I am mystified, I have written probably 100,000 lines
    of Python and never seen a thread just lock up and quit
    running. It happens on a Queue() statement so my suspicion
    is a bug. ??

    I have kludged around it by putting all the thread/queue stuff
    in the main program and import the stuff I don't want to
    distribute. But mysteries haunt your dreams, sooo...
    #!/usr/local/bin/python

    # pnetx.py

    from threading import *
    from time import *
    from Queue import Queue
    from socket import *
    import sys
    import os

    # glob is a DUMMY CLASS

    glob.listenq = Queue(1000)

    def listener():
    while 1:
    msg,addrport = listenersocket.recvfrom(BUFSIZE)
    addr = addrport[0]
    glob.listenq.put (( msg,addr),1)
    if msg == 'DIE': break

    def procmsgs():
    while 1:
    msg,addr = glob.listenq.get(1)
    if msg == 'DIE': break
    wds = msg.split(';')
    if wds[0] == 'S': recvsucc(msg); continue
    if wds[0] == 'E': recvevent(msg); continue
    if wds[0] == 'ACK': recvack(msg[4:]); continue
    if wds[0] == 'MONITOR':
    if addr not in monitorlist:
    print 'This line ALWAYS PRINTS'
    queuelist.append( Queue(0) )
    ## The above fails if this code is imported
    ## It doesn't matter whether it is imported
    ## as .py or .pyc
    ## also mq = Queue(0); queuelist.append(mq) # fails
    print 'This line ALWAYS PRINTS if i execute this source'
    print 'but NEVER PRINTS if I import '
    print 'this source from a 1 line program'
    print 'Thats weird'
    monitoron.append( 1 )
    monitorlist.append( addr )

    queuelist = [Queue(0)]


    listenersocket = socket(AF_INET,SOCK_DGRAM)
    listenersocket.bind(ListenAddr)

    procm = Thread(target=procmsgs)
    procm.start()
    listenr = Thread(target=listener)
    listenr.start()

    ## Then start a bunch of other threads and queuea.


    Don't spend a lot of time on this, not worth it.
    I just thought someone might have experienced
    something similar.
    phil, Dec 10, 2004
    #1
    1. Advertising

  2. phil

    Peter Hansen Guest

    phil wrote:
    > Uses no locks.


    It does use locks implicitly, though, since even just
    importing threading will do that, and creating a Queue
    does too.

    > I am mystified, I have written probably 100,000 lines
    > of Python and never seen a thread just lock up and quit
    > running. It happens on a Queue() statement so my suspicion
    > is a bug. ??


    You have the source to Queue.py in your standard library
    folder. Why not throw a few more print statements into
    its __init__ and see what you learn?

    > I have kludged around it by putting all the thread/queue stuff
    > in the main program and import the stuff I don't want to
    > distribute. But mysteries haunt your dreams, sooo...
    > #!/usr/local/bin/python

    [snip source]

    I cobbled together a "working" version of your code
    and ran it just fine, whether imported or run directly.
    No lockups. On Windows XP.

    Are you by any chance running on a new version of the
    Linux kernel, where the threading model has changed?
    (Or was it just RedHat 9.0?)

    I don't know the details, but I know folks have had trouble
    with this and Python... For example, I found this
    reference to the issue:
    http://groups.google.ca/groups?th=fe9a064ffeb38adc&seekm=bpi438$qta$08$-online.com#link2

    -Peter
    Peter Hansen, Dec 10, 2004
    #2
    1. Advertising

  3. On Fri, 10 Dec 2004 16:18:51 -0600, phil <> wrote:

    >And sorry I got ticked, frustrating week
    >

    threading problems can do that ;-)

    You are obviusly deeper into this than I can get from a cursory scan,
    but I'll make some general debugging comments ;-)

    > >And I could help more, being fairly experienced with
    > >threading issues and race conditions and such, but
    > >as I tried to indicate in the first place, you've
    > >provided next to no useful (IMHO) information to
    > >let anyone help you more than this

    >
    >This is about 5% of the code.
    >Uses no locks.
    >I am mystified, I have written probably 100,000 lines
    >of Python and never seen a thread just lock up and quit
    >running. It happens on a Queue() statement so my suspicion
    >is a bug. ??

    Or a once-in-a-blue-moon resource access deadlock of some kind?

    >
    >I have kludged around it by putting all the thread/queue stuff
    >in the main program and import the stuff I don't want to
    >distribute. But mysteries haunt your dreams, sooo...
    >#!/usr/local/bin/python
    >
    ># pnetx.py
    >
    >from threading import *
    >from time import *
    >from Queue import Queue
    >from socket import *

    Do you really need to import * ? Though the above should be safe import-wise
    you are setting yourself up for name-clash problems by putting so many in
    the same space. E.g., if something happened to match a misspelling typo in
    your program, you wouldn't get a name error exception. Etc., etc.

    One of the first rules of debugging is to eliminate the unnecessary ;-)
    At least there doesn't seem to be name clashes between the above (except
    starting with '_' which shouldn't get imported with *

    >>> d = {}
    >>> for imp in 'threading time Queue socket'.split():

    ... m = __import__(imp)
    ... for name in m.__dict__.keys():
    ... d.setdefault(name, []).append(imp)
    ...
    >>> for k,v in d.items():

    ... if len(v)!=1: print k,v
    ...
    _sleep ['threading', 'Queue']
    __file__ ['threading', 'Queue', 'socket']
    __all__ ['threading', 'Queue', 'socket']
    __builtins__ ['threading', 'Queue', 'socket']
    __name__ ['threading', 'time', 'Queue', 'socket']
    _time ['threading', 'Queue']
    __doc__ ['threading', 'time', 'Queue', 'socket']

    Just verifying an assumption ;-)
    OTOH, do you really need (ignoring wid and name and the first content of dir()):

    >>> dir()

    ['__builtins__', '__doc__', '__name__']
    >>> from threading import *
    >>> from time import *
    >>> from Queue import *
    >>> from socket import *
    >>> wid = 0
    >>> for name in dir():

    ... print repr(name),
    ... wid += len(repr(name))+1
    ... if wid>60:
    ... print
    ... wid = 0
    ...
    'AF_APPLETALK' 'AF_INET' 'AF_IPX' 'AF_UNSPEC' 'AI_ADDRCONFIG'
    'AI_ALL' 'AI_CANONNAME' 'AI_DEFAULT' 'AI_MASK' 'AI_NUMERICHOST'
    'AI_PASSIVE' 'AI_V4MAPPED' 'AI_V4MAPPED_CFG' 'BoundedSemaphore'
    'CAPI' 'Condition' 'EAI_ADDRFAMILY' 'EAI_AGAIN' 'EAI_BADFLAGS'
    'EAI_BADHINTS' 'EAI_FAIL' 'EAI_FAMILY' 'EAI_MAX' 'EAI_MEMORY'
    'EAI_NODATA' 'EAI_NONAME' 'EAI_PROTOCOL' 'EAI_SERVICE' 'EAI_SOCKTYPE'
    'EAI_SYSTEM' 'Empty' 'Event' 'Full' 'INADDR_ALLHOSTS_GROUP' 'INADDR_ANY'
    'INADDR_BROADCAST' 'INADDR_LOOPBACK' 'INADDR_MAX_LOCAL_GROUP'
    'INADDR_NONE' 'INADDR_UNSPEC_GROUP' 'IPPORT_RESERVED' 'IPPORT_USERRESERVED'
    'IPPROTO_GGP' 'IPPROTO_ICMP' 'IPPROTO_IDP' 'IPPROTO_IGMP' 'IPPROTO_IP'
    'IPPROTO_MAX' 'IPPROTO_ND' 'IPPROTO_PUP' 'IPPROTO_RAW' 'IPPROTO_TCP'
    'IPPROTO_UDP' 'IP_ADD_MEMBERSHIP' 'IP_DEFAULT_MULTICAST_LOOP'
    'IP_DEFAULT_MULTICAST_TTL' 'IP_DROP_MEMBERSHIP' 'IP_MAX_MEMBERSHIPS'
    'IP_MULTICAST_IF' 'IP_MULTICAST_LOOP' 'IP_MULTICAST_TTL' 'IP_OPTIONS'
    'IP_TOS' 'IP_TTL' 'Lock' 'MSG_DONTROUTE' 'MSG_OOB' 'MSG_PEEK'
    'NI_DGRAM' 'NI_MAXHOST' 'NI_MAXSERV' 'NI_NAMEREQD' 'NI_NOFQDN'
    'NI_NUMERICHOST' 'NI_NUMERICSERV' 'Queue' 'RAND_add' 'RAND_egd'
    'RAND_status' 'RLock' 'SOCK_DGRAM' 'SOCK_RAW' 'SOCK_RDM' 'SOCK_SEQPACKET'
    'SOCK_STREAM' 'SOL_IP' 'SOL_SOCKET' 'SOL_TCP' 'SOL_UDP' 'SOMAXCONN'
    'SO_ACCEPTCONN' 'SO_BROADCAST' 'SO_DEBUG' 'SO_DONTROUTE' 'SO_ERROR'
    'SO_KEEPALIVE' 'SO_LINGER' 'SO_OOBINLINE' 'SO_RCVBUF' 'SO_RCVLOWAT'
    'SO_RCVTIMEO' 'SO_REUSEADDR' 'SO_SNDBUF' 'SO_SNDLOWAT' 'SO_SNDTIMEO'
    'SO_TYPE' 'SO_USELOOPBACK' 'SSLType' 'SSL_ERROR_EOF' 'SSL_ERROR_INVALID_ERROR_CODE'
    'SSL_ERROR_SSL' 'SSL_ERROR_SYSCALL' 'SSL_ERROR_WANT_CONNECT'
    'SSL_ERROR_WANT_READ' 'SSL_ERROR_WANT_WRITE' 'SSL_ERROR_WANT_X509_LOOKUP'
    'SSL_ERROR_ZERO_RETURN' 'Semaphore' 'SocketType' 'TCP_NODELAY'
    'Thread' 'Timer' '__builtins__' '__doc__' '__name__' 'accept2dyear'
    'activeCount' 'altzone' 'asctime' 'clock' 'ctime' 'currentThread'
    'daylight' 'enumerate' 'error' 'errorTab' 'gaierror' 'getaddrinfo'
    'getdefaulttimeout' 'getfqdn' 'gethostbyaddr' 'gethostbyname'
    'gethostbyname_ex' 'gethostname' 'getnameinfo' 'getprotobyname'
    'getservbyname' 'gmtime' 'has_ipv6' 'herror' 'htonl' 'htons'
    'inet_aton' 'inet_ntoa' 'localtime' 'mktime' 'ntohl' 'ntohs'
    'setdefaulttimeout' 'setprofile' 'settrace' 'sleep' 'socket'
    'ssl' 'sslerror' 'strftime' 'strptime' 'struct_time' 'time' 'timeout'
    'timezone' 'tzname' 'wid'
    >>>


    (Hm, should have pre-tested wid+len(current) to limit width ;-)

    >import sys
    >import os
    >
    ># glob is a DUMMY CLASS

    where from? accidentally snipped?
    >
    >glob.listenq = Queue(1000)
    >
    >def listener():
    > while 1:
    > msg,addrport = listenersocket.recvfrom(BUFSIZE)
    > addr = addrport[0]

    what is the usage level on glob.listenq? It is finite 1000, so
    you could theoretically block on the put. What do you get
    if you print glob.listenq.qsize here?

    > glob.listenq.put (( msg,addr),1)
    > if msg == 'DIE': break
    >
    >def procmsgs():
    > while 1:
    > msg,addr = glob.listenq.get(1)
    > if msg == 'DIE': break

    Might be interesting to get some output when 'DIE' is recognized. Above too.

    > wds = msg.split(';')

    assert wds[0] in ['S', 'E', 'ACK', 'MONITOR'] # check assumptions. E.g.,
    # split on ';' means spaces around
    # the ';' are not eliminated
    > if wds[0] == 'S': recvsucc(msg); continue
    > if wds[0] == 'E': recvevent(msg); continue
    > if wds[0] == 'ACK': recvack(msg[4:]); continue
    > if wds[0] == 'MONITOR':
    > if addr not in monitorlist:
    > print 'This line ALWAYS PRINTS'
    > queuelist.append( Queue(0) )

    Why are you apparently creating *new* queues in this thread that you are not using
    and appending them to queuelist in the main thread? Are you testing to see how many
    can be created, or for some Queue creation bug?

    What do you get if you print 'queuelist length = %s'%len(queuelist) here?
    How often are you getting 'MONITOR'?

    > ## The above fails if this code is imported

    Imported from where? Do you mean that this code works if it
    is embedded in some larger code, and if you put an import of this
    at the top of that code it doesn't work? That would not only be
    moving the source, but also the point of invocation. I.e., the
    import *executes* what you import once, putting the result in
    the module's global name space (other than code that explicitly
    accesses other modules etc.) Do you need to wrap part of this
    in a function that you can call *after* importing the definitions?

    If your real app has multiple threads accessing a main thread queuelist
    without a lock, you might want to look closely and see if that needs
    to be a queue mechanism itself.

    > ## It doesn't matter whether it is imported
    > ## as .py or .pyc
    > ## also mq = Queue(0); queuelist.append(mq) # fails
    > print 'This line ALWAYS PRINTS if i execute this source'
    > print 'but NEVER PRINTS if I import '
    > print 'this source from a 1 line program'
    > print 'Thats weird'
    > monitoron.append( 1 )
    > monitorlist.append( addr )
    >

    Do you want all the following to be executed once at the point of importing this?

    >queuelist = [Queue(0)]

    What is this for? (oops, I hope I restored the above line correctly)
    >
    >
    >listenersocket = socket(AF_INET,SOCK_DGRAM)
    >listenersocket.bind(ListenAddr)
    >
    >procm = Thread(target=procmsgs)
    >procm.start()
    >listenr = Thread(target=listener)
    >listenr.start()
    >
    >## Then start a bunch of other threads and queuea.
    >
    >
    >Don't spend a lot of time on this, not worth it.
    >I just thought someone might have experienced
    >something similar.
    >


    Just some thoughts. HTH.


    Regards,
    Bengt Richter
    Bengt Richter, Dec 11, 2004
    #3
  4. phil

    Peter Otten Guest

    phil wrote:

    > And sorry I got ticked, frustrating week
    >
    > >And I could help more, being fairly experienced with
    > >threading issues and race conditions and such, but
    > >as I tried to indicate in the first place, you've
    > >provided next to no useful (IMHO) information to
    > >let anyone help you more than this

    >
    > This is about 5% of the code.
    > Uses no locks.


    A lock is acquired in Queue.__init__().

    > I am mystified, I have written probably 100,000 lines
    > of Python and never seen a thread just lock up and quit
    > running. It happens on a Queue() statement so my suspicion
    > is a bug. ??
    >
    > I have kludged around it by putting all the thread/queue stuff
    > in the main program and import the stuff I don't want to
    > distribute. But mysteries haunt your dreams, sooo...


    What I believe to be a minimal example:

    <freeze.py>
    import Queue
    import threading
    import time

    q = Queue.Queue(4)

    def proc():
    while True:
    q.get(1)
    Queue.Queue()
    print "YADDA"

    threading.Thread(target=proc).start()

    while True:
    print "yadda"
    q.put(None)
    time.sleep(1)
    </freeze.py>

    <freezemain.py>
    import freeze
    </freezemain.py>

    Invoking freezemain.py produces

    yadda
    yadda
    yadda
    yadda
    yadda
    yadda

    i. e. consistently q.maxsize + 2. One item about to be put, one already
    taken before Queue.Queue(). Deferring execution of the module-level code
    until after the import

    <nofreeze.py>
    import Queue
    import threading
    import time

    q = Queue.Queue(4)

    def proc():
    while True:
    q.get(1)
    Queue.Queue()
    print "YADDA"

    def start():
    threading.Thread(target=proc).start()

    while True:
    print "yadda"
    q.put(None)
    time.sleep(1)
    </nofreeze.py>

    <nofreezemain.py>
    import nofreeze
    nofreeze.start()
    </nofreezemain.py>

    and invoking nofreezemain.py produces
    yadda
    YADDA
    yadda
    YADDA
    yadda
    YADDA
    yadda
    YADDA

    apparently ad infinitum. Conclusion? None so far, though you might welcome
    the confirmation that this is not a "once in a blue moon" accident as Bengt
    Richter surmised.

    Import _is_ a sensitive phase...

    As an experiment I moved

    try:
    import thread
    except ImportError:
    import dummy_thread as thread

    from the Queue.Queue.__init__() method to the module body -- and now
    freezemain.py seems to work, too. So that would be an easy remedy, but sure
    there is a reason why that import statement is in such an unusual place?

    Peter
    Peter Otten, Dec 11, 2004
    #4
  5. phil

    Tim Peters Guest

    [Peter Otten]
    > What I believe to be a minimal example:
    >
    > <freeze.py>
    > import Queue
    > import threading
    > import time
    >
    > q = Queue.Queue(4)
    >
    > def proc():
    > while True:
    > q.get(1)
    > Queue.Queue()
    > print "YADDA"
    >
    > threading.Thread(target=proc).start()
    >
    > while True:
    > print "yadda"
    > q.put(None)
    > time.sleep(1)
    > </freeze.py>
    >
    > <freezemain.py>
    > import freeze
    > </freezemain.py>


    CPython has an internal, reentrant import lock. When a thread does an
    import, it acquires this lock, and the lock remains held until that
    import is complete. As a consequence, no *other* thread can do an
    import (it blocks waiting to obtain the internal import lock) until
    the original import completes. So until "import freeze" returns in
    the main thread, no other thread can do an import.

    Partly for that reason, it's generally a Horrible Idea to start a
    thread as a side effect of importing a module. That's what freeze.py
    does, and you get the expected deadlock as a result. The main thread
    is hung waiting for "import freeze" to return, and the spawned thread
    is hung at an import in Queue.__init__() waiting for the main thread
    to release the import lock.

    > Invoking freezemain.py produces
    >
    > yadda
    > yadda
    > yadda
    > yadda
    > yadda
    > yadda
    >
    > i. e. consistently q.maxsize + 2. One item about to be put, one
    > already taken before Queue.Queue(). Deferring execution of the
    > module-level code until after the import
    >
    > <nofreeze.py>
    > import Queue
    > import threading
    > import time
    >
    > q = Queue.Queue(4)
    >
    > def proc():
    > while True:
    > q.get(1)
    > Queue.Queue()
    > print "YADDA"
    >
    > def start():
    > threading.Thread(target=proc).start()
    >
    > while True:
    > print "yadda"
    > q.put(None)
    > time.sleep(1)
    > </nofreeze.py>
    >
    > <nofreezemain.py>
    > import nofreeze
    > nofreeze.start()
    > </nofreezemain.py>
    >
    > and invoking nofreezemain.py produces
    > yadda
    > YADDA
    > yadda
    > YADDA
    > yadda
    > YADDA
    > yadda
    > YADDA
    >
    > apparently ad infinitum.


    Right. Note that this is the same reason threaded tests in Python's
    standard regression suite define a 'test_main()' function, called by
    the regrtest.py driver after import of the test's module completes.
    It's generally suicidal to start a thread as a side effect of an
    import.

    ....

    > Import _is_ a sensitive phase...


    It's quite easy to avoid thread problems in imports: never start a
    thread as a side effect of importing, and you'll never get a deadlock
    due to importing.

    > As an experiment I moved
    >
    > try:
    > import thread
    > except ImportError:
    > import dummy_thread as thread
    >
    > from the Queue.Queue.__init__() method to the module body --
    > and now freezemain.py seems to work, too. So that would be an
    > easy remedy, but sure there is a reason why that import
    > statement is in such an unusual place?


    I think you'd have to ask Brett (who did most of the work on
    dummy_thread and dummy_threading). It doesn't really matter, though:
    it's a general truth that starting a thread as a side effect of
    importing is a recipe for deadlock, and hacking specific methods and
    functions to avoid imports just moves the problem around. It's not a
    goal that anything in the standard Python library cater to bad thread
    practice here (the bad thread practice being, again, starting a thread
    as a side effect of importing).
    Tim Peters, Dec 12, 2004
    #5
  6. Op 2004-12-12, Tim Peters schreef <>:
    >
    > I think you'd have to ask Brett (who did most of the work on
    > dummy_thread and dummy_threading). It doesn't really matter, though:
    > it's a general truth that starting a thread as a side effect of
    > importing is a recipe for deadlock, and hacking specific methods and
    > functions to avoid imports just moves the problem around. It's not a
    > goal that anything in the standard Python library cater to bad thread
    > practice here (the bad thread practice being, again, starting a thread
    > as a side effect of importing).


    I don't see why starting a thread as a side effect of importing is
    bad thread practice. Sure python doesn't cater for it, but IMO
    that seems to be python failing.

    --
    Antoon Pardon
    Antoon Pardon, Dec 13, 2004
    #6
  7. phil

    Tim Peters Guest

    [Antoon Pardon]
    > I don't see why starting a thread as a side effect of importing is
    > bad thread practice. Sure python doesn't cater for it, but IMO
    > that seems to be python failing.


    Obviously, it's bad practice in Python because it can lead to
    deadlocks in Python. It's nearly tautological. Import is an
    executable statement in Python, not, e.g., as in many other languages,
    a declaration directed at the system linker. With that power comes
    opportunities to shoot yourself, although they're generally easy to
    avoid. Come up with a practical design that doesn't have this
    limitation, and then perhaps your characterization of the current
    design as "a failing" would be both credible and constructive.

    Apart from that, ya, I do think it would *uisually* be poor practice
    to start a thread as a side effect of importing anyway. It's too
    mysterious, and IME has caused trouble even when it didn't lead to
    deadlocks. The fundamental purpose of import in Python is to add
    useful names to the importer's namespace, and users of a module
    generally think of it as doing no more than that.

    Note that the OP's example had a module that, upon the first attempt
    to import it, ran an infinite loop (even if it hadn't deadlocked), and
    it's clearly severe abuse of import's purpose.to write a module M such
    that "import M" *never* returns. Indeed, that's the other half of how
    deadlock occurs: not only that the imported module spawn a thread as
    a side effect of importing, but also that the imported module refuse
    to allow the import to complete.

    The current design actually supports spawning all the threads you like
    as a side effect of importing, provided you ensure also that the
    import ompletes. The easiest way to avoid trouble remains not to
    spawn threads as a side effect of importing to begin with, although a
    programmer determined to demonstrate their bad taste <wink> can easily
    enough make it work.
    Tim Peters, Dec 13, 2004
    #7
  8. Op 2004-12-13, Tim Peters schreef <>:
    > [Antoon Pardon]
    >> I don't see why starting a thread as a side effect of importing is
    >> bad thread practice. Sure python doesn't cater for it, but IMO
    >> that seems to be python failing.

    >
    > Obviously, it's bad practice in Python because it can lead to
    > deadlocks in Python.


    By that argument any use of locks is bad practice because it
    can lead to deadlock.

    > It's nearly tautological. Import is an
    > executable statement in Python, not, e.g., as in many other languages,
    > a declaration directed at the system linker. With that power comes
    > opportunities to shoot yourself, although they're generally easy to
    > avoid. Come up with a practical design that doesn't have this
    > limitation, and then perhaps your characterization of the current
    > design as "a failing" would be both credible and constructive.


    If a car model has cranky brakes, I think I can call that a failing
    even without having the ability to come up with a pratical design
    that doesn's has those limitations.

    I judge a language by what it can and cannot do, not by my ability
    to correct the things I perceive as failings. For all I know python
    may have taken some design decisions that might have seen perfectly
    logical but now prohibit a a practical design that doesn't have this
    limitation. I don't see why something like that would make this
    any less a failing then when a practical design was easy in the
    current implemenation.

    > Apart from that, ya, I do think it would *uisually* be poor practice
    > to start a thread as a side effect of importing anyway. It's too
    > mysterious, and IME has caused trouble even when it didn't lead to
    > deadlocks. The fundamental purpose of import in Python is to add
    > useful names to the importer's namespace, and users of a module
    > generally think of it as doing no more than that.


    Importing a module in general also does some kind of initialisation.
    If starting a thread is a logical thing to do in this initialisation
    fase I see nothing wrong with it.

    > Note that the OP's example had a module that, upon the first attempt
    > to import it, ran an infinite loop (even if it hadn't deadlocked), and
    > it's clearly severe abuse of import's purpose.to write a module M such
    > that "import M" *never* returns. Indeed, that's the other half of how
    > deadlock occurs: not only that the imported module spawn a thread as
    > a side effect of importing, but also that the imported module refuse
    > to allow the import to complete.


    Well I'll agree here. An import that has as a side effect that the
    import doesn't return is bad practice.

    > The current design actually supports spawning all the threads you like
    > as a side effect of importing, provided you ensure also that the
    > import ompletes.


    Well in that case I don't have any problems with it. The perceived
    failing was because of only knowing part of it based on what I had
    read in this thread.

    > The easiest way to avoid trouble remains not to
    > spawn threads as a side effect of importing to begin with, although a
    > programmer determined to demonstrate their bad taste <wink> can easily
    > enough make it work.


    Well then probably I have a bad taste. I have though of designs in which
    it seemed very natural to have a thread started as part of the
    initialisation in a module. Other limitations of python didn't make
    it workable but in priciple I saw nothing wrong with doing it.

    --
    Antoon Pardon
    Antoon Pardon, Dec 14, 2004
    #8
  9. phil

    Steve Holden Guest

    Antoon Pardon wrote:

    > Op 2004-12-13, Tim Peters schreef <>:
    >
    >>[Antoon Pardon]
    >>
    >>>I don't see why starting a thread as a side effect of importing is
    >>>bad thread practice. Sure python doesn't cater for it, but IMO
    >>>that seems to be python failing.

    >>
    >>Obviously, it's bad practice in Python because it can lead to
    >>deadlocks in Python.

    >
    >
    > By that argument any use of locks is bad practice because it
    > can lead to deadlock.
    >

    Not at all. You mentioned locks, not Tim. The reason it can lead to
    deadlocks is because import, being an executable statement, must
    terminate correctly in order to return control to the module that
    executed the import statement.
    >
    >>It's nearly tautological. Import is an
    >>executable statement in Python, not, e.g., as in many other languages,
    >>a declaration directed at the system linker. With that power comes
    >>opportunities to shoot yourself, although they're generally easy to
    >>avoid. Come up with a practical design that doesn't have this
    >>limitation, and then perhaps your characterization of the current
    >>design as "a failing" would be both credible and constructive.

    >
    >
    > If a car model has cranky brakes, I think I can call that a failing
    > even without having the ability to come up with a pratical design
    > that doesn's has those limitations.
    >

    But in fact your situation is more closely analogous to a customer who's
    bought a car that can be stopped by pressing on the brake pedal now
    complaining that sideways pressure on the brake pedal doesn;t affect the
    car's motion.

    > I judge a language by what it can and cannot do, not by my ability
    > to correct the things I perceive as failings. For all I know python
    > may have taken some design decisions that might have seen perfectly
    > logical but now prohibit a a practical design that doesn't have this
    > limitation. I don't see why something like that would make this
    > any less a failing then when a practical design was easy in the
    > current implemenation.
    >

    All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
    as the result of a specific call to programmed functionality rather than
    as the side effect of an import. The reason for this is due to the
    semantics of the import statement. If you perceive that as a failing
    then you'd be better rewarded by an attempt to modify your perceptions.

    I've always found "don't argue with Tim about Python" to be a useful
    rule of thumb. He's wrong much less often than I am. I suspect he's also
    wrong much less often than you ;-)
    >
    >>Apart from that, ya, I do think it would *uisually* be poor practice
    >>to start a thread as a side effect of importing anyway. It's too
    >>mysterious, and IME has caused trouble even when it didn't lead to
    >>deadlocks. The fundamental purpose of import in Python is to add
    >>useful names to the importer's namespace, and users of a module
    >>generally think of it as doing no more than that.

    >
    >
    > Importing a module in general also does some kind of initialisation.
    > If starting a thread is a logical thing to do in this initialisation
    > fase I see nothing wrong with it.
    >

    But the reason, I suspect, that it's being suggested this is bad design
    is because it would be tricky (in any language) to ensure that the
    satisfactory conclusion of the import didn't depend on successful thread
    startup (and possibly termination).

    Plus, of course, in Python the side-effect of the import (creating the
    thread) can happen precisely once, because the module body is executed
    precisely once no matter how many times it's imported. That's not really
    the issue, however.

    >
    >>Note that the OP's example had a module that, upon the first attempt
    >>to import it, ran an infinite loop (even if it hadn't deadlocked), and
    >>it's clearly severe abuse of import's purpose.to write a module M such
    >>that "import M" *never* returns. Indeed, that's the other half of how
    >>deadlock occurs: not only that the imported module spawn a thread as
    >>a side effect of importing, but also that the imported module refuse
    >>to allow the import to complete.

    >
    >
    > Well I'll agree here. An import that has as a side effect that the
    > import doesn't return is bad practice.
    >

    But that's precisely the risk you run when starting up threads!
    >
    >>The current design actually supports spawning all the threads you like
    >>as a side effect of importing, provided you ensure also that the
    >>import ompletes.

    >
    >
    > Well in that case I don't have any problems with it. The perceived
    > failing was because of only knowing part of it based on what I had
    > read in this thread.
    >
    >
    >>The easiest way to avoid trouble remains not to
    >>spawn threads as a side effect of importing to begin with, although a
    >>programmer determined to demonstrate their bad taste <wink> can easily
    >>enough make it work.

    >
    >
    > Well then probably I have a bad taste. I have though of designs in which
    > it seemed very natural to have a thread started as part of the
    > initialisation in a module. Other limitations of python didn't make
    > it workable but in priciple I saw nothing wrong with doing it.
    >

    Bearing in mind that module initialisation is always one-off, relying on
    import to trigger such complex behavior is probably a design that will
    mislead users into false expectations.

    regards
    Steve
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
    Steve Holden, Dec 14, 2004
    #9
  10. Op 2004-12-14, Steve Holden schreef <>:
    > Antoon Pardon wrote:
    >
    >> Op 2004-12-13, Tim Peters schreef <>:
    >>
    >>>[Antoon Pardon]
    >>>
    >>>>I don't see why starting a thread as a side effect of importing is
    >>>>bad thread practice. Sure python doesn't cater for it, but IMO
    >>>>that seems to be python failing.
    >>>
    >>>Obviously, it's bad practice in Python because it can lead to
    >>>deadlocks in Python.

    >>
    >>
    >> By that argument any use of locks is bad practice because it
    >> can lead to deadlock.
    >>

    > Not at all. You mentioned locks, not Tim.


    That is beside the point. The argument was that starting a thread
    as a side effect of importing is bad practice because it can lead to
    deadlock. This suggest that a general condition for something
    to be bad practice is if that something can lead to deadlock.
    Locks are in that case.

    >>>It's nearly tautological. Import is an
    >>>executable statement in Python, not, e.g., as in many other languages,
    >>>a declaration directed at the system linker. With that power comes
    >>>opportunities to shoot yourself, although they're generally easy to
    >>>avoid. Come up with a practical design that doesn't have this
    >>>limitation, and then perhaps your characterization of the current
    >>>design as "a failing" would be both credible and constructive.

    >>
    >>
    >> If a car model has cranky brakes, I think I can call that a failing
    >> even without having the ability to come up with a pratical design
    >> that doesn's has those limitations.
    >>

    > But in fact your situation is more closely analogous to a customer who's
    > bought a car that can be stopped by pressing on the brake pedal now
    > complaining that sideways pressure on the brake pedal doesn;t affect the
    > car's motion.


    You will have to explain how you come by that analogy.

    >> I judge a language by what it can and cannot do, not by my ability
    >> to correct the things I perceive as failings. For all I know python
    >> may have taken some design decisions that might have seen perfectly
    >> logical but now prohibit a a practical design that doesn't have this
    >> limitation. I don't see why something like that would make this
    >> any less a failing then when a practical design was easy in the
    >> current implemenation.
    >>

    > All that Tim was suggesting is that it's MORE SENSIBLE to start a thread
    > as the result of a specific call to programmed functionality rather than
    > as the side effect of an import. The reason for this is due to the
    > semantics of the import statement. If you perceive that as a failing
    > then you'd be better rewarded by an attempt to modify your perceptions.


    > I've always found "don't argue with Tim about Python" to be a useful
    > rule of thumb. He's wrong much less often than I am. I suspect he's also
    > wrong much less often than you ;-)


    With all respect I find that lousy advise. I don't mind that it will turn
    out I'll be wrong most of the time. But I'll probably will have gained
    a better understanding by the responses to my argument than by merely
    accepting the word of Tim.

    [ ... ]

    >>>Note that the OP's example had a module that, upon the first attempt
    >>>to import it, ran an infinite loop (even if it hadn't deadlocked), and
    >>>it's clearly severe abuse of import's purpose.to write a module M such
    >>>that "import M" *never* returns. Indeed, that's the other half of how
    >>>deadlock occurs: not only that the imported module spawn a thread as
    >>>a side effect of importing, but also that the imported module refuse
    >>>to allow the import to complete.

    >>
    >>
    >> Well I'll agree here. An import that has as a side effect that the
    >> import doesn't return is bad practice.
    >>

    > But that's precisely the risk you run when starting up threads!


    Now you are confusing me. Is it a problem of an import that doesn't
    return of is it a case of a race condition where the import has to
    return in a timely fashion?

    --
    Antoon Pardon
    Antoon Pardon, Dec 15, 2004
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. phil

    thread/queue bug

    phil, Dec 10, 2004, in forum: Python
    Replies:
    1
    Views:
    293
    Peter Hansen
    Dec 10, 2004
  2. phil

    thread/queue bug

    phil, Dec 10, 2004, in forum: Python
    Replies:
    2
    Views:
    362
    Steve Holden
    Dec 12, 2004
  3. phil

    thread/queue bug

    phil, Dec 10, 2004, in forum: Python
    Replies:
    1
    Views:
    295
    Peter Hansen
    Dec 10, 2004
  4. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    678
    Russell Warren
    Jun 27, 2006
  5. Kris
    Replies:
    0
    Views:
    478
Loading...

Share This Page