socket's strange behavior with subprocesses

[Jane Austine]

Running Python 2.3 on Win XP

It seems like socket is working interdependently with subprocesses of
the process which created socket.

------------------------------------
#the server side

import socket
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('localhost',9000))
s.listen(5)
z=s.accept()
import os
f=os.popen('notepad.exe') #notepad appears on the screen
z[0]

z[0].recv(6) 'foobar'
z[0].send('hello world')

11

#the client sidehello world

------------------------------
Now when the client requests to recv 1024 bytes, and since there is no
more to read from the socket it blocks.

#client side
and the server side tries to close the socket:

#server side

z[0].close()
#yes, it seems to have worked.

Alas, the client side doesn't wake up! It doesn't wake up unless the
notepad is exited first; only after that, 'Connection reset by peer'
is raised. What does the socket has to do with subprocesses?

 

[Irmen de Jong]

and the server side tries to close the socket:

#server side

z[0].close()
#yes, it seems to have worked.

Alas, the client side doesn't wake up! It doesn't wake up unless the
notepad is exited first; only after that, 'Connection reset by peer'
is raised. What does the socket has to do with subprocesses?

Nothing, I guess... try to shutdown the socket explicitly before closing it:


z[0].shutdown(2)
z[0].close()

does that work?

--Irmen

 

[Colin Brown]

Running Python 2.3 on Win XP

It seems like socket is working interdependently with subprocesses of
the process which created socket.
and the server side tries to close the socket: ......
Alas, the client side doesn't wake up! It doesn't wake up unless the
notepad is exited first; only after that, 'Connection reset by peer'
is raised. What does the socket has to do with subprocesses?

Hi Jane

I think it may be tied up with the way subprocesses are made under Windows.
Below is a copy of part of a post I made some months back when inexplicable
file-locking was causing me problems. It took me weeks to track down the
reason.

The Python default subprocess inherits open handles from the parent.
Your options are:
1. Open the subprocess before the socket.
2. Create the subprocess using win32 API primitives (this is what I had to
do).
Let me know if want details.

Colin Brown
PyNZ

----------------------------------------------------------------------------
------
I have been struggling to solve why an occasional "Permission Denied" error
popped up from the following code fragment in one of my Win2K program
threads:

....
input_file = preprocess(raw_file)
os.system('third_party input_file output_file > error_file')
if os.path.exists(saved_file):
os.remove(saved_file)
os.rename(input_file,saved_file)
....

The error occurs on the os.rename(). The third_party executable was closing
input_file before terminating and anyway the subshell process has finished
before the os.rename is called! Most baffling.

With the aid of handle.exe from www.sysinternals.com I finally resolved the
problem. To see the problem at first hand try the following script:

import time, thread, os

def rm(file):
print 'delete x.x'
os.remove(file)

def wait():
if os.name == 'nt':
os.system('pause')
elif os.name == 'posix':
os.system('sleep 3')
print 'end wait'

print 'create x.x'
f = open('x.x','w')
thread.start_new_thread(wait,())
time.sleep(1)
print '\nclose x.x'
f.close()
rm('x.x')

Although this works fine on Linux, I get a Permission Denied error on
Windows. I surmise that an os.system call uses a C fork to create the
subshell - an exact duplicate of the current process environment including
OPEN FILE HANDLES. Because the os.system call has not returned before the
os.remove is called a "Permission Denied" error occurs. Quite simple really.

Now going back to my original problem, I had other threads doing os.system
calls. When one of these calls happens during the preprocess file write of
my above thread AND takes longer to complete than the os.system call in the
above thread then the file will still be open and the os.rename will return
the Permission Denied error.

 

[Jane Austine]

Hi Jane

I think it may be tied up with the way subprocesses are made under Windows.
Below is a copy of part of a post I made some months back when inexplicable
file-locking was causing me problems. It took me weeks to track down the
reason.

The Python default subprocess inherits open handles from the parent.
Your options are:
1. Open the subprocess before the socket.
2. Create the subprocess using win32 API primitives (this is what I had to
do).
Let me know if want details.

Colin Brown
PyNZ

Thank you very much, first of all.

I tried win32 API primitives for creating subprocesses: win32all's
CreateProcess. I used the Process class in winprocess.py in the
"demos" directory. However, it didn't work with sockets perfectly.

Say, a socket server python script is launched via CreateProcess. It
then launches sub-processes. And I kill(via TerminateProcess) the
socket server process. The subprocesses remain alive(as expected). I
try to connect to the 'dead server port'. I expect almost immediate
"connect refused" error, but it doesn't come up until the subprocesses
are all gone and client hangs forever.

Jane

 

[Colin Brown]

"Colin Brown" <[email protected]> wrote in message ....
I tried win32 API primitives for creating subprocesses: win32all's
CreateProcess. I used the Process class in winprocess.py in the
"demos" directory. However, it didn't work with sockets perfectly.

Say, a socket server python script is launched via CreateProcess. It
then launches sub-processes. And I kill(via TerminateProcess) the
socket server process. The subprocesses remain alive(as expected). I
try to connect to the 'dead server port'. I expect almost immediate
"connect refused" error, but it doesn't come up until the subprocesses
are all gone and client hangs forever.

Jane

If I am interpreting what you are saying here correctly you have:

Main_process
=> [create_process]
=> Sub_process1
socket_server_connection
subprocess2 (of subprocess1) started

Sub_process1 terminated, but socket connection held until subprocess2
terminated.

This is what I would expect based on my findings. Subprocess2 has inherited
the socket handle when it was created (assuming you used os.system,
os.spawn* or os.popen*). You would have to use the correct incantation of
create_process to launch subprocess2.

I have attached the code I used in place of os.system.

Colin

--[Win32.py]---------------------------------------------------------------
# perform equivalent of os.system without open file handles

import win32process,win32event

def system(cmd):
handles = win32process.CreateProcess \
(None,cmd,None,None,0,0,None,None,win32process.STARTUPINFO())
status = win32event.WaitForSingleObject(handles[0],win32event.INFINITE)
if status == win32event.WAIT_ABANDONED:
raise 'win32.system WAIT_ABANDONED'
elif status == win32event.WAIT_FAILED:
raise 'win32.system WAIT_FAILED'
elif status == win32event.WAIT_IO_COMPLETION:
raise 'win32.system WAIT_IO_COMPLETION'
elif status == win32event.WAIT_OBJECT_0:
pass
elif status == win32event.WAIT_TIMEOUT:
raise 'win32.system WAIT_TIMEOUT'
else:
raise 'win32.system - unknown event status = '+str(status)