Using multiprocessing from a Windows service

[Volodya]

Hi all,

I think I've found a small bug with multiprocessing package on
Windows. If you try to start a multiprocessing.Process from a Python-
based Windows service, the child process will fail to run. When
running the parent process as a regular Python program, everything
works as expected.

I've tracked the problem down to how main_path is prepared in
multiprocessing.forking.get_preparation_data() (lines 370-377):

def get_preparation_data(name):
[...skipped a few lines...]
if not WINEXE:
main_path = getattr(sys.modules['__main__'], '__file__', None)
if not main_path and sys.argv[0] not in ('', '-c'):
main_path = sys.argv[0]
if main_path is not None:
if not os.path.isabs(main_path) and \
process.ORIGINAL_DIR is not
None:
main_path = os.path.join(process.ORIGINAL_DIR,
main_path)
d['main_path'] = os.path.normpath(main_path)
return d

When the program is running as a Windows service, but is not packaged
into a single executable, main_path will become the path to the
service executable (typically, pythonservice.exe). When this data
makes it to the child process, the prepare() function will treat
main_path as a path to a python module, and will try to import it.
This causes it to fail.

My quick-and-dirty solution was to check in get_preparation_data() if
main_path ends with '.exe', and if it does, to not pass it at all.
This solves the problem in my case, but perhaps there's a better way
to fix this? Here is my version of get_preparation_data():

def get_preparation_data(name):
'''
Return info about parent needed by child to unpickle process
object
'''
from .util import _logger, _log_to_stderr

d = dict(
name=name,
sys_path=sys.path,
sys_argv=sys.argv,
log_to_stderr=_log_to_stderr,
orig_dir=process.ORIGINAL_DIR,
authkey=process.current_process().authkey,
)

if _logger is not None:
d['log_level'] = _logger.getEffectiveLevel()

if not WINEXE:
main_path = getattr(sys.modules['__main__'], '__file__', None)
if not main_path and sys.argv[0] not in ('', '-c'):
main_path = sys.argv[0]
if main_path is not None:
if not os.path.isabs(main_path) and \
process.ORIGINAL_DIR is not
None:
main_path = os.path.join(process.ORIGINAL_DIR,
main_path)
if not main_path.endswith('.exe'):
d['main_path'] = os.path.normpath(main_path)

return d

 

[Mark Hammond]

Hi all,

I think I've found a small bug with multiprocessing package on
Windows.

I'd actually argue its a bug in pythonservice.exe - it should set
sys.argv[] to resemble a normal python process with argv[0] being the
script. I'll fix it...

Cheers,

Mark

 

[Mark Hammond]

Hi all,

I think I've found a small bug with multiprocessing package on
Windows.

I'd actually argue its a bug in pythonservice.exe - it should set
sys.argv[] to resemble a normal python process with argv[0] being the
script. I'll fix it...

Actually it appears I spoke too soon:

* A bug in pywin32 exists such that when you use 'debug' on a service,
the argv reflected the full argv of the application, including the
'-debug' portion of the command-line. However, even if that is fixed,
the next argument is actually the name of the service (as declared in
the .py file for the service), not the .py module itself. Thus, I could
make argv a little more sane in this case, but still the initial problem
would remain as argv[0] would still not be a .py file.

* When the service is started by windows itself, there are usually zero
additional command-line arguments. If there *are* arguments, they are
likely to be the string the user entered via control panel as a special
case (Windows doesn't actually remember service args - they are used
once and discarded). It is important we continue to expose whatever
argv we actually got from Windows to the service code.

So unfortunately I don't think I can change pythonservice to resolve the
issue you reported.

Cheers,

Mark

 

[Volodymyr Orlenko]

Hi all,

I think I've found a small bug with multiprocessing package on
Windows.

I'd actually argue its a bug in pythonservice.exe - it should set
sys.argv[] to resemble a normal python process with argv[0] being the
script. I'll fix it...

Actually it appears I spoke too soon:

* A bug in pywin32 exists such that when you use 'debug' on a service,
the argv reflected the full argv of the application, including the
'-debug' portion of the command-line. However, even if that is fixed,
the next argument is actually the name of the service (as declared in
the .py file for the service), not the .py module itself. Thus, I
could make argv a little more sane in this case, but still the initial
problem would remain as argv[0] would still not be a .py file.

* When the service is started by windows itself, there are usually
zero additional command-line arguments. If there *are* arguments,
they are likely to be the string the user entered via control panel as
a special case (Windows doesn't actually remember service args - they
are used once and discarded). It is important we continue to expose
whatever argv we actually got from Windows to the service code.

So unfortunately I don't think I can change pythonservice to resolve
the issue you reported.

Thanks Mark, this makes perfect sense.

In the patch I submitted, I simply check if the name of the supposed
module ends with ".exe". It works fine for my case, but maybe this is
too general. Is there a chance that a Python module would end in ".exe"?
If so, maybe we should check specifically for "pythonservice.exe". But
then, if someone renames the executable (as I did, because I wanted to
see meaningful service names in the process list), the patch will not
work. Maybe there's another way to fix the forking module?

 

[James Mills]

In the patch I submitted, I simply check if the name of the supposed module
ends with ".exe". It works fine for my case, but maybe this is too general.
Is there a chance that a Python module would end in ".exe"? If so, maybe we
should check specifically for "pythonservice.exe". But then, if someone
renames the executable (as I did, because I wanted to see meaningful service
names in the process list), the patch will not work. Maybe there's another
way to fix the forking module?

I believe the best way to fix this is to fix the underlying
issue that Mark has pointed out (monkey-patching mp won't do).

--JamesMills

 

[Volodymyr Orlenko]

[...] Maybe there's another
way to fix the forking module?

I believe the best way to fix this is to fix the underlying
issue that Mark has pointed out (monkey-patching mp won't do).

But according to Mark's second message in this thread, there's no way to
make pythonservice.exe set its sys.argv[0] to a python module -- it uses
sys.argv for debug parameters in debug mode, and for custom service
parameters in normal mode.

 

[Mark Hammond]

In the patch I submitted, I simply check if the name of the supposed
module ends with ".exe". It works fine for my case, but maybe this is
too general. Is there a chance that a Python module would end in ".exe"?

IIRC, py2exe may create executables where sys.argv[0] is the executable
itself. Maybe if we consider and handle both these cases a patch to mp
might be looked upon in a better light...

Cheers,

Mark