Different behavior with multiprocessing

[Tomas Kotal]

Hi all.

Recently I discovered a strange behavior with multiprocessing library and call to function os.system (a different behavior under Linux and Windows to be more specific). I have this simple testing script:

############################################
import sys
import os
from multiprocessing import Process

def do_fork(cmd):
ret = os.system(cmd)
# print result of call
print ret
os._exit(ret)

if __name__ == "__main__":
cmds = [ "dir", "xy" ]

procs = []
for cmd in cmds:
proc = Process(target=do_fork, args=(cmd, ))
proc.start()
procs.append( proc )

for proc in procs:
proc.join()
print "exitcode: %d" % proc.exitcode

print "ok"
############################################

This script just starts 2 processes. Each process executes one command in system shell and exits setting it's exit code same as exit code of a system call. There are 2 commands: "dir" (which works fine on Windows and Linux) and "xy" (which is supposed to fail under both systems). The strange thing is that when I run this script under Windows I get this output:

0
1
exitcode: 0
exitcode: 1
ok

The first 0 and 1 are results of os.system call which are printed from child processes. Rest of lines are printed from main process. This is expected output.

But when I run same script on Linux, what I get is this:

0
32512
exitcode: 0
exitcode: 0
ok

Although the second command fails and returns exit code 32512, the exit code from process in parent process gives me 0. I tried to change the script to use fork() instead of Process but the result was same.

Can anybody explain me what's the problem here?

(I use Python 2.6 on both Windows and Linux machines)

 

[Chris Angelico]

But when I run same script on Linux, what I get is this:

0
32512

Under Unix, the return value from os.system() encodes more than one
piece of information:

http://docs.python.org/2/library/os.html#os.system
http://docs.python.org/2/library/os.html#os.wait

32512 is 127*256, meaning that the shell exited with return code 127
when given an unrecognized command.

ChrisA

 

[Tomas Kotal]

Under Unix, the return value from os.system() encodes more than one

piece of information:



http://docs.python.org/2/library/os.html#os.system

http://docs.python.org/2/library/os.html#os.wait



32512 is 127*256, meaning that the shell exited with return code 127

when given an unrecognized command.



ChrisA

Well the point is not the difference between return values of os.system under Windows and Linux.

The point is that I set the exist code of child process with os._exit() which works fine on Windows but on Linux the process.exitcode returns always 0.

What is very strange is, that if I set return code to some other value, like:

....
os._exit( len(cmd) )
....

Then I got exit codes 3 and 2 under Linux, which is correct. But when I do something like this:

....
ret = os.system(cmd)
print ret # this print real value, like 32512
os._exit(ret) # exitcode of this child process will be 0 on Linux
....

It doesn't work, which seems very strange to me.

 

[Tomas Kotal]

Under Unix, the return value from os.system() encodes more than one

piece of information:



http://docs.python.org/2/library/os.html#os.system

http://docs.python.org/2/library/os.html#os.wait



32512 is 127*256, meaning that the shell exited with return code 127

when given an unrecognized command.



ChrisA

Well the point is not the difference between return values of os.system under Windows and Linux.

The point is that I set the exist code of child process with os._exit() which works fine on Windows but on Linux the process.exitcode returns always 0.

What is very strange is, that if I set return code to some other value, like:

....
os._exit( len(cmd) )
....

Then I got exit codes 3 and 2 under Linux, which is correct. But when I do something like this:

....
ret = os.system(cmd)
print ret # this print real value, like 32512
os._exit(ret) # exitcode of this child process will be 0 on Linux
....

It doesn't work, which seems very strange to me.

 

[Tomas Kotal]

Dne pondělí, 4. března 2013 15:31:41 UTC+1 Tomas Kotal napsal(a):

Hi all.



Recently I discovered a strange behavior with multiprocessing library andcall to function os.system (a different behavior under Linux and Windows to be more specific). I have this simple testing script:



############################################

import sys

import os

from multiprocessing import Process



def do_fork(cmd):

ret = os.system(cmd)

# print result of call

print ret

os._exit(ret)



if __name__ == "__main__":

cmds = [ "dir", "xy" ]



procs = []

for cmd in cmds:

proc = Process(target=do_fork, args=(cmd, ))

proc.start()

procs.append( proc )



for proc in procs:

proc.join()

print "exitcode: %d" % proc.exitcode



print "ok"

############################################



This script just starts 2 processes. Each process executes one command insystem shell and exits setting it's exit code same as exit code of a system call. There are 2 commands: "dir" (which works fine on Windows and Linux)and "xy" (which is supposed to fail under both systems). The strange thingis that when I run this script under Windows I get this output:



0

1

exitcode: 0

exitcode: 1

ok



The first 0 and 1 are results of os.system call which are printed from child processes. Rest of lines are printed from main process. This is expected output.



But when I run same script on Linux, what I get is this:



0

32512

exitcode: 0

exitcode: 0

ok



Although the second command fails and returns exit code 32512, the exit code from process in parent process gives me 0. I tried to change the scriptto use fork() instead of Process but the result was same.



Can anybody explain me what's the problem here?



(I use Python 2.6 on both Windows and Linux machines)

Seems like I found the problem: os._exit probably takes as parametr unsigned char, so it uses as error code whatever value it gets modulo 256:

os._exit(1) # process.exitcode == 1
os._exit(255) # process.exitcode == 255
os._exit(256) # process.exitcode == 0
os._exit(257) # process.exitcode == 1
os._exit(32512) # process.exitcode == 0

So on Linux it's necesary to call something like this:
os._exit( os.system(cmd) >> 8 )

Because the first byte of return value on Linux is number of signal which kills the process and the second one is actual exit code.

http://docs.python.org/2/library/os.html#os.wait

 

[Chris Angelico]

Seems like I found the problem: os._exit probably takes as parametr unsigned char, so it uses as error code whatever value it gets modulo 256:

os._exit(1) # process.exitcode == 1
os._exit(255) # process.exitcode == 255
os._exit(256) # process.exitcode == 0
os._exit(257) # process.exitcode == 1
os._exit(32512) # process.exitcode == 0

So on Linux it's necesary to call something like this:
os._exit( os.system(cmd) >> 8 )

Because the first byte of return value on Linux is number of signal which kills the process and the second one is actual exit code.

Yep. I had a reply part-written but you beat me to it! That is indeed
what you need if you want to chain return values.

However, why are you using os._exit? Check out the note here:

http://docs.python.org/2/library/os.html#os._exit

ChrisA