calling functions across threads

[Steven Bethard]

I'm playing around with some threading stuff right now, and I'm having a
little trouble calling a function from one thread that affects another.
Here's my setup:

py> import os, threading, time
py> def write(file_in, input_lines):
.... for line in input_lines:
.... time.sleep(0.5)
.... file_in.write(line)
.... file_in.flush()
.... file_in.close()
....
py> def read(file_out, output_list):
.... while True:
.... line = file_out.readline()
.... if not line:
.... break
.... output_list.append(line)
....
py> def runthreads(lst):
.... file_in, file_out, file_err = os.popen3('cat')
.... write_thread = threading.Thread(
.... target=write, args=(file_in,
.... ['%s\n' % x for x in range(10)]))
.... read_thread = threading.Thread(target=read,
.... args=(file_out, lst))
.... write_thread.start()
.... read_thread.start()
.... write_thread.join()
.... read_thread.join()
....

Basically, I start one thread to read and one thread to write (from a
os.pipe). This all works fine for me:

py> lst = []
py> runthreads(lst)
py> lst
['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

I run into a problem though when I try to call an update method every
time I read a line:

py> class updatinglist(list):
.... def __init__(self, updater):
.... super(updatinglist, self).__init__()
.... self.updater = updater
.... def append(self, item):
.... super(updatinglist, self).append(item)
.... self.updater(len(self))
....
py> def update(i):
.... print i
....
py> lst = updatinglist(update)
py> runthreads(lst)
1
2
3
4
5
6
7
8
9
10
py> lst
['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

I get the correct output, but if you run this yourself, you'll see that
the numbers 1 through 10 aren't printed in sync with the writes (i.e.
every half second); they're all printed at the end. Could someone
explain to me why this happens, and how (if possible) I can get the
numbers printed in sync with the appends to the list?

Thanks,

Steve

 

[Thomas Rast]

I get the correct output, but if you run this yourself, you'll see
that the numbers 1 through 10 aren't printed in sync with the writes
(i.e. every half second); they're all printed at the end. Could
someone explain to me why this happens, and how (if possible) I can
get the numbers printed in sync with the appends to the list?

I tried your code, and got the expected behaviour, i.e. the numbers
are printed every half second. Maybe you have a buffering problem?

$ python2.4 -V
Python 2.4
$ uname -a
Linux thomas 2.6.9 #6 Sun Dec 19 17:45:53 CET 2004 i686 GNU/Linux

- Thomas

 

[Steven Bethard]

I tried your code, and got the expected behaviour, i.e. the numbers
are printed every half second. Maybe you have a buffering problem?

$ python2.4 -V
Python 2.4
$ uname -a
Linux thomas 2.6.9 #6 Sun Dec 19 17:45:53 CET 2004 i686 GNU/Linux

FWIW, I'm using Python 2.4 on a Windows XP box. But it looks like my
problems were due to using PythonWin. When I tried the same code at the
command prompt python, it worked just fine.

Anyone know why PythonWin would do this? I originally ran into this
problem using simliar code in a module that I used with the Ellogon
(www.ellogon.org) framework. Maybe if I can solve the problem for
PythonWin, I can translate that into a solution for Ellogon too...

Steve

 

[Fernando Perez]

I get the correct output, but if you run this yourself, you'll see that
the numbers 1 through 10 aren't printed in sync with the writes (i.e.
every half second); they're all printed at the end. Could someone
explain to me why this happens, and how (if possible) I can get the
numbers printed in sync with the appends to the list?

This is just a shot in the dark, as I'm quite ignorant of threading details.
But what happens if you try adding a sys.stdout.flush() call after the print
statement in your custom update() method? It may just be a flushing problem
what makes the output appear out of sync...

Cheers,

f

 

[Steven Bethard]

Steven Bethard wrote:




This is just a shot in the dark, as I'm quite ignorant of threading details.
But what happens if you try adding a sys.stdout.flush() call after the print
statement in your custom update() method? It may just be a flushing problem
what makes the output appear out of sync...

Strangely enough, that causes PythonWin to hang... Why that would be
true, I have no idea...

Steve

 

[Fernando Perez]

Strangely enough, that causes PythonWin to hang... Why that would be
true, I have no idea...

Mmh. I wouldn't be surprised if under pythonwin, sys.stdout is not the true
python sys.stdout. Check the following:

sys.stdout is sys.__stdout__

The answer is probably false. In that case, they may have implemented some
incomplete object whose flush method is broken, or something similar. I can't
confirm, as I don't have windows access, so this is just a guess.

Cheers,

f

 

[Steven Bethard]

Steven Bethard wrote:




Mmh. I wouldn't be surprised if under pythonwin, sys.stdout is not the true
python sys.stdout. Check the following:

sys.stdout is sys.__stdout__

The answer is probably false. In that case, they may have implemented some
incomplete object whose flush method is broken, or something similar. I can't
confirm, as I don't have windows access, so this is just a guess.

Just to verify, yes, the answer is False:

py> import sys
py> sys.stdout is sys.__stdout__
False

Is there a list to ask PythonWin specific questions somewhere, or should
I just wait for a PythonWin expert around here?

Steve

 

[Steve Holden]

Just to verify, yes, the answer is False:

py> import sys
py> sys.stdout is sys.__stdout__
False

Is there a list to ask PythonWin specific questions somewhere, or should
I just wait for a PythonWin expert around here?

Steve

There's a mailing list at (e-mail address removed) - it's listed on
www.python.org should you choose to subscribe, and the volume isn't
generally high (maybe 5-10 messages per day).

regards
Steve

 

[It's me]

I haven't play with the thread stuff in Python (yet) but in general terms
(from a C mind), one should not expect read/write actions to be sequential
across threads. I would assume the Python threads eventually goes back to
some system calls for thread handling. If that were the case, you should
not be surprised at all that the I/O sequences appear to be quite
unpredictable.

If you absolutely, positively wants them to come out in a certain way, you
need to build in additionally serialization mechanisims in your code (like
use semaphores and stuff).

I'm playing around with some threading stuff right now, and I'm having a
little trouble calling a function from one thread that affects another.
Here's my setup:

py> import os, threading, time
py> def write(file_in, input_lines):
... for line in input_lines:
... time.sleep(0.5)
... file_in.write(line)
... file_in.flush()
... file_in.close()
...
py> def read(file_out, output_list):
... while True:
... line = file_out.readline()
... if not line:
... break
... output_list.append(line)
...
py> def runthreads(lst):
... file_in, file_out, file_err = os.popen3('cat')
... write_thread = threading.Thread(
... target=write, args=(file_in,
... ['%s\n' % x for x in range(10)]))
... read_thread = threading.Thread(target=read,
... args=(file_out, lst))
... write_thread.start()
... read_thread.start()
... write_thread.join()
... read_thread.join()
...

Basically, I start one thread to read and one thread to write (from a
os.pipe). This all works fine for me:

py> lst = []
py> runthreads(lst)
py> lst
['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

I run into a problem though when I try to call an update method every
time I read a line:

py> class updatinglist(list):
... def __init__(self, updater):
... super(updatinglist, self).__init__()
... self.updater = updater
... def append(self, item):
... super(updatinglist, self).append(item)
... self.updater(len(self))
...
py> def update(i):
... print i
...
py> lst = updatinglist(update)
py> runthreads(lst)
1
2
3
4
5
6
7
8
9
10
py> lst
['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

I get the correct output, but if you run this yourself, you'll see that
the numbers 1 through 10 aren't printed in sync with the writes (i.e.
every half second); they're all printed at the end. Could someone
explain to me why this happens, and how (if possible) I can get the
numbers printed in sync with the appends to the list?

Thanks,

Steve