Interesting Thread Gotcha

Discussion in 'Python' started by Hendrik van Rooyen, Jan 15, 2008.

  1. I thought I would share this nasty little gotcha with the group.

    Consider the following code fragment:

    <start>
    print 'starting kbd thread'
    keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))
    print 'starting main loop'
    error = Mainloop(s,port_q,active_q_list)
    <end>

    It produces, as output, the following:

    starting kbd thread
    we get here - a

    It does not print 'starting main loop', the Mainloop routine
    is never executed, and no exceptions are raised.

    Here is the offending routine that seems to capture the control:

    <start>
    def kbd_driver(out_q,in_q):
    """
    thread to look for keyboard input and to put it on the queue out_q
    also looks for replies on in_q and prints them
    """

    kbdname = '/dev/stdin'

    kbd = open(kbdname,'r+',1) # Reading, line buffered

    unblock(kbd) # Call the magic to unblock keyboard
    print 'we get here - a'
    while True:

    try:
    d = kbd.readline() # see if any kbd input
    except:
    IOError
    try:
    msg=in_q.get(block=False)
    except Queue.Empty:
    time.sleep(0.1)
    continue
    print msg
    time.sleep(0.1)
    continue
    d = d.rstrip() # get rid of line feed
    out_q.put([d + '\r',in_q]) # add a carriage return and return q and send
    to port
    <end>


    The unblock is a routine that unblocks a port using fcntl - it
    is not the problem. In case you don't believe me, here it is:

    def unblock(f):
    """Given file 'f', sets its unblock flag to true."""

    fcntl.fcntl(f.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)

    I will post the solution tomorrow when I read my mail,
    if no one has spotted it by then.

    - Hendrik
     
    Hendrik van Rooyen, Jan 15, 2008
    #1
    1. Advertising

  2. Hendrik van Rooyen

    Dan Guest

    On Jan 15, 10:07 am, "Hendrik van Rooyen" <>
    wrote:
    > I thought I would share this nasty little gotcha with the group.
    >
    > Consider the following code fragment:
    >
    > <start>
    > print 'starting kbd thread'
    > keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))
    > print 'starting main loop'
    > error = Mainloop(s,port_q,active_q_list)
    > <end>
    >
    > It produces, as output, the following:
    >
    > starting kbd thread
    > we get here - a
    >
    > It does not print 'starting main loop', the Mainloop routine
    > is never executed, and no exceptions are raised.
    >
    > Here is the offending routine that seems to capture the control:
    >
    > <start>
    > def kbd_driver(out_q,in_q):
    > """
    > thread to look for keyboard input and to put it on the queue out_q
    > also looks for replies on in_q and prints them
    > """
    >
    > kbdname = '/dev/stdin'
    >
    > kbd = open(kbdname,'r+',1) # Reading, line buffered
    >
    > unblock(kbd) # Call the magic to unblock keyboard
    > print 'we get here - a'
    > while True:
    >
    > try:
    > d = kbd.readline() # see if any kbd input
    > except:
    > IOError
    > try:
    > msg=in_q.get(block=False)
    > except Queue.Empty:
    > time.sleep(0.1)
    > continue
    > print msg
    > time.sleep(0.1)
    > continue
    > d = d.rstrip() # get rid of line feed
    > out_q.put([d + '\r',in_q]) # add a carriage return and return q and send
    > to port
    > <end>
    >
    > The unblock is a routine that unblocks a port using fcntl - it
    > is not the problem. In case you don't believe me, here it is:
    >
    > def unblock(f):
    > """Given file 'f', sets its unblock flag to true."""
    >
    > fcntl.fcntl(f.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
    >
    > I will post the solution tomorrow when I read my mail,
    > if no one has spotted it by then.
    >
    > - Hendrik


    >>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))


    Needs to be
    >>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))


    Commas are important!

    -Dan
     
    Dan, Jan 15, 2008
    #2
    1. Advertising

  3. "Dan" <the,,,ail.com> wrote:


    > >>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))

    >
    > Needs to be
    > >>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))

    >
    > Commas are important!
    >
    > -Dan


    Absolutely! - well spotted!

    As the first correct respondent, you win the freedom to spend a week in
    Naboomspruit at your own expense.

    It would have been nice, however, to have gotten something like:

    TypeError - This routine needs a tuple.

    instead of the silent in line calling of the routine in question,
    while failing actually to start a new thread.

    It seems to act no different from plain old:

    kbd_driver (port_q,kbd_q)

    Is it worth the trouble of learning how to submit a bug report?

    - Hendrik
     
    Hendrik van Rooyen, Jan 16, 2008
    #3
  4. Hendrik van Rooyen

    Duncan Booth Guest

    "Hendrik van Rooyen" <> wrote:

    > It would have been nice, however, to have gotten something like:
    >
    > TypeError - This routine needs a tuple.
    >
    > instead of the silent in line calling of the routine in question,
    > while failing actually to start a new thread.


    Given that the start_new_thread function never actually got called, what
    code exactly do you expect to complain about the absence of a tuple?

    >
    > It seems to act no different from plain old:
    >
    > kbd_driver (port_q,kbd_q)
    >
    > Is it worth the trouble of learning how to submit a bug report?


    On your own code? There doesn't appear to be a bug in anyone else's code
    here.
     
    Duncan Booth, Jan 16, 2008
    #4
  5. Hendrik van Rooyen wrote:
    > Absolutely! - well spotted!


    This is no threading problem at all; not even a syntax problem. If
    you don't know exactly what start_new_thread and kbd_driver
    functions do it's impossible to tell if your code does what is
    intended.

    > It would have been nice, however, to have gotten something like:
    >
    > TypeError - This routine needs a tuple.
    >
    > instead of the silent in line calling of the routine in question,
    > while failing actually to start a new thread.


    Exactly which part of the code should give you this warning?

    > Is it worth the trouble of learning how to submit a bug report?


    For your problem not, IMHO, as a bug report for it will be closed
    quickly.

    Regards,


    Björn

    --
    BOFH excuse #330:

    quantum decoherence
     
    Bjoern Schliessmann, Jan 16, 2008
    #5
  6. Hendrik van Rooyen wrote:

    > "Dan" <the,,,ail.com> wrote:
    >
    >
    >> >>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))

    >>
    >> Needs to be
    >> >>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))

    >>
    >> Commas are important!
    >>
    >> -Dan

    >
    > Absolutely! - well spotted!
    >
    > As the first correct respondent, you win the freedom to spend a week in
    > Naboomspruit at your own expense.
    >
    > It would have been nice, however, to have gotten something like:
    >
    > TypeError - This routine needs a tuple.
    >
    > instead of the silent in line calling of the routine in question,
    > while failing actually to start a new thread.


    You can't prevent the silent inline-calling - otherwise, how would you do
    this:

    def compute_thread_target():
    def target():
    pass
    return target

    thread.start_new_thread(compute_thread_target())


    Of course start_new_thread could throw an error if it got nothing callable
    as first argument. No idea why it doesn't.

    Diez
     
    Diez B. Roggisch, Jan 16, 2008
    #6
  7. Hendrik van Rooyen

    Dan Guest

    On Jan 16, 11:06 am, "Diez B. Roggisch" <> wrote:
    > Hendrik van Rooyen wrote:
    > > "Dan" <the,,,ail.com> wrote:

    >
    > >> >>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))

    >
    > >> Needs to be
    > >> >>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))

    >
    > >> Commas are important!

    >
    > >> -Dan

    >
    > > Absolutely! - well spotted!

    >
    > > As the first correct respondent, you win the freedom to spend a week in
    > > Naboomspruit at your own expense.

    >
    > > It would have been nice, however, to have gotten something like:

    >
    > > TypeError - This routine needs a tuple.

    >
    > > instead of the silent in line calling of the routine in question,
    > > while failing actually to start a new thread.

    >
    > You can't prevent the silent inline-calling - otherwise, how would you do
    > this:
    >
    > def compute_thread_target():
    > def target():
    > pass
    > return target
    >
    > thread.start_new_thread(compute_thread_target())
    >
    > Of course start_new_thread could throw an error if it got nothing callable
    > as first argument. No idea why it doesn't.
    >
    > Diez


    Of course, in his case, having start_new_thread throw an error
    wouldn't have helped, since he went into an infinite loop while
    evaluating the parameters for start_new_thread.

    Would it be possible to have pychecker (or some such) warn that there
    is an insufficient parameter count to start_new_thread? I guess that
    would require knowing the type of thread. . .

    -Dan
     
    Dan, Jan 16, 2008
    #7
  8. Hendrik van Rooyen

    Dan Guest

    On Jan 16, 1:33 pm, "Diez B. Roggisch" <> wrote:
    > Dan schrieb:
    >
    >
    >
    > > On Jan 16, 11:06 am, "Diez B. Roggisch" <> wrote:
    > >> Hendrik van Rooyen wrote:
    > >>> "Dan" <the,,,ail.com> wrote:
    > >>>>>>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))
    > >>>> Needs to be
    > >>>>>>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))
    > >>>> Commas are important!
    > >>>> -Dan
    > >>> Absolutely! - well spotted!
    > >>> As the first correct respondent, you win the freedom to spend a week in
    > >>> Naboomspruit at your own expense.
    > >>> It would have been nice, however, to have gotten something like:
    > >>> TypeError - This routine needs a tuple.
    > >>> instead of the silent in line calling of the routine in question,
    > >>> while failing actually to start a new thread.
    > >> You can't prevent the silent inline-calling - otherwise, how would you do
    > >> this:

    >
    > >> def compute_thread_target():
    > >> def target():
    > >> pass
    > >> return target

    >
    > >> thread.start_new_thread(compute_thread_target())

    >
    > >> Of course start_new_thread could throw an error if it got nothing callable
    > >> as first argument. No idea why it doesn't.

    >
    > >> Diez

    >
    > > Of course, in his case, having start_new_thread throw an error
    > > wouldn't have helped, since he went into an infinite loop while
    > > evaluating the parameters for start_new_thread.

    >
    > > Would it be possible to have pychecker (or some such) warn that there
    > > is an insufficient parameter count to start_new_thread? I guess that
    > > would require knowing the type of thread. . .

    >
    > What has this to do with the second argument? It's perfectly legal to
    > have a function as thread-target that takes no arguments at all, so
    > enforcing a second argument wouldn't be helpful - all it would do is to
    > force all developers that don't need an argument tuple to pass the empty
    > tuple. So there was no insufficient argument count.
    >
    > And none of these would solve the underlying problem that in python
    > expressions are evaluated eagerly. Changing that would mean that you end
    > up with a totally new language.
    >
    > the only thing that could help to a certain extend would be static
    > types. Which we don't want here :)
    >
    > Diez


    It doesn't seem to be legal in my version of python (or the doc):

    >>> import thread
    >>> def bat():

    print "hello"


    >>> thread.start_new_thread(bat)


    Traceback (most recent call last):
    File "<pyshell#12>", line 1, in <module>
    thread.start_new_thread(bat)
    TypeError: start_new_thread expected at least 2 arguments, got 1
    >>> thread.start_new_thread(bat, ())

    2256hello


    >>>


    -Dan
     
    Dan, Jan 16, 2008
    #8
  9. Dan schrieb:
    > On Jan 16, 1:33 pm, "Diez B. Roggisch" <> wrote:
    >> Dan schrieb:
    >>
    >>
    >>
    >>> On Jan 16, 11:06 am, "Diez B. Roggisch" <> wrote:
    >>>> Hendrik van Rooyen wrote:
    >>>>> "Dan" <the,,,ail.com> wrote:
    >>>>>>>>> keyboard_thread = thread.start_new_thread(kbd_driver (port_q,kbd_q))
    >>>>>> Needs to be
    >>>>>>>>> keyboard_thread = thread.start_new_thread(kbd_driver, (port_q,kbd_q))
    >>>>>> Commas are important!
    >>>>>> -Dan
    >>>>> Absolutely! - well spotted!
    >>>>> As the first correct respondent, you win the freedom to spend a week in
    >>>>> Naboomspruit at your own expense.
    >>>>> It would have been nice, however, to have gotten something like:
    >>>>> TypeError - This routine needs a tuple.
    >>>>> instead of the silent in line calling of the routine in question,
    >>>>> while failing actually to start a new thread.
    >>>> You can't prevent the silent inline-calling - otherwise, how would you do
    >>>> this:
    >>>> def compute_thread_target():
    >>>> def target():
    >>>> pass
    >>>> return target
    >>>> thread.start_new_thread(compute_thread_target())
    >>>> Of course start_new_thread could throw an error if it got nothing callable
    >>>> as first argument. No idea why it doesn't.
    >>>> Diez
    >>> Of course, in his case, having start_new_thread throw an error
    >>> wouldn't have helped, since he went into an infinite loop while
    >>> evaluating the parameters for start_new_thread.
    >>> Would it be possible to have pychecker (or some such) warn that there
    >>> is an insufficient parameter count to start_new_thread? I guess that
    >>> would require knowing the type of thread. . .

    >> What has this to do with the second argument? It's perfectly legal to
    >> have a function as thread-target that takes no arguments at all, so
    >> enforcing a second argument wouldn't be helpful - all it would do is to
    >> force all developers that don't need an argument tuple to pass the empty
    >> tuple. So there was no insufficient argument count.
    >>
    >> And none of these would solve the underlying problem that in python
    >> expressions are evaluated eagerly. Changing that would mean that you end
    >> up with a totally new language.
    >>
    >> the only thing that could help to a certain extend would be static
    >> types. Which we don't want here :)
    >>
    >> Diez

    >
    > It doesn't seem to be legal in my version of python (or the doc):
    >
    >>>> import thread
    >>>> def bat():

    > print "hello"
    >
    >
    >>>> thread.start_new_thread(bat)

    >
    > Traceback (most recent call last):
    > File "<pyshell#12>", line 1, in <module>
    > thread.start_new_thread(bat)
    > TypeError: start_new_thread expected at least 2 arguments, got 1
    >>>> thread.start_new_thread(bat, ())

    > 2256hello


    Ah, I thought it was optional, as in the threading.Thread(target=...,
    args=....)-version. Sorry for not looking that up.

    Then you'd might stand a chance that pychecker can find such a situation
    - but of course not on a general level, as in the above - that would
    only work with type-annotations.



    Diez
     
    Diez B. Roggisch, Jan 16, 2008
    #9
  10. "Duncan Booth" <dunc...d.invalid> wrote:

    > Given that the start_new_thread function never actually got called, what
    > code exactly do you expect to complain about the absence of a tuple?


    I don't understand this assertion.

    I thought that start_new_thread was called with a missing comma in
    its argument list, which had the effect that I am complaining about.

    Putting the comma in place solved the problem, without any other
    changes, so why do you say that start_new_thread was not called?

    - Hendrik
     
    Hendrik van Rooyen, Jan 17, 2008
    #10
  11. "Bjoern Schliessmann" <usenet-ourmet.com> wrote:

    Hendrik van Rooyen wrote:
    > Absolutely! - well spotted!


    This is no threading problem at all; not even a syntax problem. If
    you don't know exactly what start_new_thread and kbd_driver
    functions do it's impossible to tell if your code does what is
    intended.

    > It would have been nice, however, to have gotten something like:
    >
    > TypeError - This routine needs a tuple.
    >
    > instead of the silent in line calling of the routine in question,
    > while failing actually to start a new thread.


    Exactly which part of the code should give you this warning?

    I am obviously missing something.

    My understanding is that, to start a new thread, one does:

    NewThreadID = thread.start_new_thread(NameOfRoutineToStart,
    (ArgumentToCall_it_with,secondArg,Etc))

    This calls start_new_thread with the name and the arguments to pass.

    If one omits the comma, then start_new_thread is surely stilled called,
    but with an argument that is now a call to the routine in question, which
    somehow causes the problem.

    So start_new_thread is the code that that is executed, with a bad set of
    arguments - one thing, (a call to a routine) instead of two things -
    a routine and a tuple of arguments.

    Everywhere else in Python if you give a routine the incorrect number of
    arguments, you get an exception. Why not here?

    - Hendrik
     
    Hendrik van Rooyen, Jan 17, 2008
    #11
  12. Diez B. Roggisch <> wrote:
    >Of course start_new_thread could throw an error if it got nothing callable
    >as first argument. No idea why it doesn't.


    It does:

    >>> thread.start_new_thread(None, None)

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    TypeError: first arg must be callable

    --
    \S -- -- http://www.chaos.org.uk/~sion/
    "Frankly I have no feelings towards penguins one way or the other"
    -- Arthur C. Clarke
    her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
     
    Sion Arrowsmith, Jan 17, 2008
    #12
  13. "Diez B. Roggisch" <dee,,eb.de> wrote:
    > Hendrik van Rooyen wrote:
    > > It would have been nice, however, to have gotten something like:
    > >
    > > TypeError - This routine needs a tuple.
    > >
    > > instead of the silent in line calling of the routine in question,
    > > while failing actually to start a new thread.

    >
    > You can't prevent the silent inline-calling - otherwise, how would you do
    > this:
    >
    > def compute_thread_target():
    > def target():
    > pass
    > return target
    >
    > thread.start_new_thread(compute_thread_target())
    >
    >
    > Of course start_new_thread could throw an error if it got nothing callable
    > as first argument. No idea why it doesn't.


    Thanks - got it, I think. Doesn't mean I like it, though:

    >>> a = 42
    >>> b = 24
    >>> def do_something(c,d):

    print c
    print d

    >>> do_something(a,b)

    42
    24

    >>> def harmless():

    return a

    >>> def evil():

    while True:
    pass

    >>> do_something(a)

    Traceback (most recent call last):
    File "<pyshell#15>", line 1, in ?
    do_something(a)
    TypeError: do_something() takes exactly 2 arguments (1 given)
    >>> do_something(harmless())

    Traceback (most recent call last):
    File "<pyshell#17>", line 1, in ?
    do_something(harmless())
    TypeError: do_something() takes exactly 2 arguments (1 given)
    >>>do_something(evil())



    This hangs and needs OS intervention to kill it - and there is also just
    one argument, not two.

    Looks like the arguments are handled one by one without validation
    till the end. Lets see:

    >>> do_something(a,b,harmless())

    Traceback (most recent call last):
    File "<pyshell#18>", line 1, in ?
    do_something(a,b,harmless())
    TypeError: do_something() takes exactly 2 arguments (3 given)

    So far, so good.

    >>>do_something(a,b,evil())


    This also hangs - the third, extra argument is actually called!

    Are you all sure this is not a buglet?

    - Hendrik
     
    Hendrik van Rooyen, Jan 17, 2008
    #13
  14. Hendrik van Rooyen

    Peter Otten Guest

    Hendrik van Rooyen wrote:

    > "Bjoern Schliessmann" <usenet-ourmet.com> wrote:
    >
    > Hendrik van Rooyen wrote:
    >> Absolutely! - well spotted!

    >
    > This is no threading problem at all; not even a syntax problem. If
    > you don't know exactly what start_new_thread and kbd_driver
    > functions do it's impossible to tell if your code does what is
    > intended.
    >
    >> It would have been nice, however, to have gotten something like:
    >>
    >> TypeError - This routine needs a tuple.
    >>
    >> instead of the silent in line calling of the routine in question,
    >> while failing actually to start a new thread.

    >
    > Exactly which part of the code should give you this warning?
    >
    > I am obviously missing something.
    >
    > My understanding is that, to start a new thread, one does:
    >
    > NewThreadID = thread.start_new_thread(NameOfRoutineToStart,
    > (ArgumentToCall_it_with,secondArg,Etc))
    >
    > This calls start_new_thread with the name and the arguments to pass.
    >
    > If one omits the comma, then start_new_thread is surely stilled called,
    > but with an argument that is now a call to the routine in question, which
    > somehow causes the problem.
    >
    > So start_new_thread is the code that that is executed, with a bad set of
    > arguments - one thing, (a call to a routine) instead of two things -
    > a routine and a tuple of arguments.
    >
    > Everywhere else in Python if you give a routine the incorrect number of
    > arguments, you get an exception. Why not here?


    Python always evaluates the function's arguments first. The check for the
    correct number of arguments is part of the call and therefore done
    afterwards:

    >>> def f(x): print x

    ....
    >>> f(f(1), f(2), f(3))

    1
    2
    3
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: f() takes exactly 1 argument (3 given)

    So if one of the arguments takes forever to calculate you will never see
    the TypeError:

    >>> def g(x):

    .... print x
    .... import time
    .... while 1: time.sleep(1)
    ....
    >>> f(f(1), g(2), f(3))

    1
    2
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 4, in g
    KeyboardInterrupt # I hit Ctrl-C

    Peter
     
    Peter Otten, Jan 17, 2008
    #14
  15. Hendrik van Rooyen

    Peter Otten Guest

    Hendrik van Rooyen wrote:

    > "Duncan Booth" <dunc...d.invalid> wrote:
    >
    >> Given that the start_new_thread function never actually got called, what
    >> code exactly do you expect to complain about the absence of a tuple?

    >
    > I don't understand this assertion.
    >
    > I thought that start_new_thread was called with a missing comma in
    > its argument list, which had the effect that I am complaining about.
    >
    > Putting the comma in place solved the problem, without any other
    > changes, so why do you say that start_new_thread was not called?


    Well, when kbd_driver() is called the kbd_q queue is probably empty, and
    as kbd_driver() runs in the main thread, who could ever put something into
    that queue? The function will therefore never terminate.

    Peter
     
    Peter Otten, Jan 17, 2008
    #15
  16. "Dan" <ther,,,ail.com> wrote:


    > Would it be possible to have pychecker (or some such) warn that there
    > is an insufficient parameter count to start_new_thread? I guess that
    > would require knowing the type of thread. . .


    I think this is the hub of the thing - its not only start_new_thread, but
    the way that parameters are evaluated before being counted, generally.

    See my reply to Diez's post

    - Hendrik
     
    Hendrik van Rooyen, Jan 18, 2008
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Roedy Green

    ZipFile.entries gotcha

    Roedy Green, Sep 26, 2003, in forum: Java
    Replies:
    9
    Views:
    624
    Phil...
    Sep 29, 2003
  2. VisionSet

    JCert & JLS gotcha

    VisionSet, Oct 6, 2003, in forum: Java
    Replies:
    0
    Views:
    356
    VisionSet
    Oct 6, 2003
  3. Siemel Naran

    Re: An Interesting Gotcha

    Siemel Naran, Apr 1, 2004, in forum: C++
    Replies:
    1
    Views:
    341
    Leor Zolman
    Apr 1, 2004
  4. Kenneth McDonald

    Interesting little "gotcha" with generators

    Kenneth McDonald, Dec 22, 2005, in forum: Python
    Replies:
    5
    Views:
    332
  5. Shafik
    Replies:
    5
    Views:
    328
    Diez B. Roggisch
    Oct 5, 2007
Loading...

Share This Page