Can multiprocessing.Process() use a different version of the interpreter?

[Scott Pigman]

I want to use the 64 bit version of python but run some code inside a
spawned process that uses the 32 bit version. Is there an official
way to do this, or can anybody make a recommendation about the
cleanest way to do this? If someone has a better solution to the
problem I describe below, I'd appreciate hearing that too. I'm using
python 2.7, but could go to 3.x if that has better support for this.

My program interacts with two different commercial libraries by using
ctypes to execute functions inside their DLLs. One library only
exists in a 32 bit version. The second library has both 64 and 32 bit
versions. The code was initially developed for a 32 bit platform, but
now I need to support a user on a 64 bit machine as well. I've
learned that ctypes can only interact with libraries which match the
version of python that is running --- 32 bit python can use 32 bit
DLLs, 64 bit python can use 64 bit DLLs, so when I'm running on a 64
bit machine with one 64 bit library and the other at 32 bits, I can't
use the 32 bit library.

I believe that the solution is to have both win32 and x64 versions of
python installed, and then execute the win32 code inside of a new
process which uses the win32 version of Python. I'm already using
multiprocessing to run that code inside a new process because the two
libraries have conflicts and need to run in separate memory space. If
I could just specify that I want the new process to use a different
version of the interpreter I think I'd be all set, but I don't see a
way of doing that.

 

[Robert Kern]

I want to use the 64 bit version of python but run some code inside a
spawned process that uses the 32 bit version. Is there an official
way to do this, or can anybody make a recommendation about the
cleanest way to do this? If someone has a better solution to the
problem I describe below, I'd appreciate hearing that too. I'm using
python 2.7, but could go to 3.x if that has better support for this.

My program interacts with two different commercial libraries by using
ctypes to execute functions inside their DLLs. One library only
exists in a 32 bit version. The second library has both 64 and 32 bit
versions. The code was initially developed for a 32 bit platform, but
now I need to support a user on a 64 bit machine as well. I've
learned that ctypes can only interact with libraries which match the
version of python that is running --- 32 bit python can use 32 bit
DLLs, 64 bit python can use 64 bit DLLs, so when I'm running on a 64
bit machine with one 64 bit library and the other at 32 bits, I can't
use the 32 bit library.

I believe that the solution is to have both win32 and x64 versions of
python installed, and then execute the win32 code inside of a new
process which uses the win32 version of Python. I'm already using
multiprocessing to run that code inside a new process because the two
libraries have conflicts and need to run in separate memory space. If
I could just specify that I want the new process to use a different
version of the interpreter I think I'd be all set, but I don't see a
way of doing that.

You cannot use a Process() to do this, but you should be able to use Managers or
the other tools in the multiprocessing packages for communicating between the
processes.

http://docs.python.org/library/multiprocessing#module-multiprocessing.managers

There may be hidden assumptions in the code, though. I've never tried it myself.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Robert Kern]

I believe that the solution is to have both win32 and x64 versions of
python installed, and then execute the win32 code inside of a new
process which uses the win32 version of Python. I'm already using
multiprocessing to run that code inside a new process because the two
libraries have conflicts and need to run in separate memory space. If
I could just specify that I want the new process to use a different
version of the interpreter I think I'd be all set, but I don't see a
way of doing that.

My previous statement notwithstanding, there is a way to hack this. The
windows-specific implementation of the class multiprocessing.forking.Popen uses
a global variable, _python_exe, to determine the executable to use. Instead, you
could subclass Popen to use whatever executable you like and subclass Process to
use this Popen. See the source files process.py and forking.py in the
multiprocessing package. It should be straightforward to attempt, but I don't
know for sure that it will work.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Scott Pigman]

My previous statement notwithstanding, there is a way to hack this. The
windows-specific implementation of the class multiprocessing.forking.Popen uses
a global variable, _python_exe, to determine the executable to use. Instead, you

What I've been looking at since I first posted is that that
_python.exe variable is built from sys.exec_prefix. So, I think that
I can make it work by resetting sys.exec_prefix to the install dir of
the 32 bit version before creating the new Process().

 

[Scott Pigman]

_python.exe variable is built from sys.exec_prefix.  So, I think that

sorry, _python_exe.

 

[Robert Kern]

sorry, _python_exe.

Personally, I think that subclassing is cleaner, but if you must monkeypatch,
then you should monkeypatch the multiprocessing.forking._python_exe variable,
not sys.exec_prefix.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 

[Scott Pigman]

Personally, I think that subclassing is cleaner, but if you must monkeypatch,
then you should monkeypatch the multiprocessing.forking._python_exe variable,
not sys.exec_prefix.

thanks, good point.