Doing readline in a thread from a popen4('rsync ...') stream blocks when the stream ends.

[Rasmusson, Lars]

Hi,

I have a problem doing I/O in a python thread.

I want a thread to execute a command using
os.popen, read its input until the end, and
then print 'DONE' and terminate.

However, the thread hangs when it reaches
the end of the stream.

The command that I want to do is 'rsync'
which is used to transfer files between hosts.
Rsync uses ssh to transfer the data securely.

Here is a small program that demonstrates
the problem:

------START-------
#!/usr/bin/python2.3
# -*- python -*-

# This program demonstrates a case when Python locks up.
# It runs the program 'rsync' and reads its output.
# When this is done inside a thread, the program locks up
# while doing the last i.readline()

# To try the program, make sure that you can ssh into localhost
# without password prompt, and that /etc/hosts exists
#

fileToGet = '/etc/hosts'
cmd='rsync -v -e ssh localhost:%s file.txt' % fileToGet

def do():
import os
(o,i)=os.popen4(cmd)
readAll(i)

def readAll(i):
x=i.readline()
while x:
print x
x=i.readline() # this is where the program locks up
print 'DONE'

from threading import Thread

class RunInThread(Thread):
def __init__(self,run,daemon=True):
Thread.__init__(self)
self.setDaemon(daemon)
self.run = run
self.start()

print '\nFirst we run the command outside of a thread. This works fine.\n'

do()

print '\nNow we will run the command inside a python thread. '
print 'The program will never print DONE, nor finish.\n'

RunInThread(do,daemon=False)

-------END-------


This problem occurs with rsync, so one guess is that it
is related to the interaction with python's threads and
and rsync or ssh.

One message I google'd up mentioned that non-standard
sighandlers could be a problem when running rsync from perl,
but I have not had any luck there.

I have also tried pexpect and pty.fork as alternatives to popen,
but with no luck.

I have the same problem on both Windows and Linux platforms.
It occurs on many versions of python.


I'd be very grateful for any suggestions/hints/solutions
to this problem!


Thanks!
Lars

 

[popov]

Hi,

I have a problem doing I/O in a python thread.

I want a thread to execute a command using
os.popen, read its input until the end, and
then print 'DONE' and terminate.

However, the thread hangs when it reaches
the end of the stream.

You can have a look here:
http://sourceforge.net/tracker/?group_id=5470&atid=105470, looking for
'popen'.
requestID #566037 may be of interest, for eg.