Can multiprocessing.Process() use a different version of the interpreter?

Discussion in 'Python' started by Scott Pigman, Oct 4, 2010.

  1. Scott Pigman

    Scott Pigman Guest

    I want to use the 64 bit version of python but run some code inside a
    spawned process that uses the 32 bit version. Is there an official
    way to do this, or can anybody make a recommendation about the
    cleanest way to do this? If someone has a better solution to the
    problem I describe below, I'd appreciate hearing that too. I'm using
    python 2.7, but could go to 3.x if that has better support for this.

    My program interacts with two different commercial libraries by using
    ctypes to execute functions inside their DLLs. One library only
    exists in a 32 bit version. The second library has both 64 and 32 bit
    versions. The code was initially developed for a 32 bit platform, but
    now I need to support a user on a 64 bit machine as well. I've
    learned that ctypes can only interact with libraries which match the
    version of python that is running --- 32 bit python can use 32 bit
    DLLs, 64 bit python can use 64 bit DLLs, so when I'm running on a 64
    bit machine with one 64 bit library and the other at 32 bits, I can't
    use the 32 bit library.

    I believe that the solution is to have both win32 and x64 versions of
    python installed, and then execute the win32 code inside of a new
    process which uses the win32 version of Python. I'm already using
    multiprocessing to run that code inside a new process because the two
    libraries have conflicts and need to run in separate memory space. If
    I could just specify that I want the new process to use a different
    version of the interpreter I think I'd be all set, but I don't see a
    way of doing that.
    Scott Pigman, Oct 4, 2010
    #1
    1. Advertising

  2. Scott Pigman

    Robert Kern Guest

    Re: Can multiprocessing.Process() use a different version of theinterpreter?

    On 10/4/10 11:39 AM, Scott Pigman wrote:
    > I want to use the 64 bit version of python but run some code inside a
    > spawned process that uses the 32 bit version. Is there an official
    > way to do this, or can anybody make a recommendation about the
    > cleanest way to do this? If someone has a better solution to the
    > problem I describe below, I'd appreciate hearing that too. I'm using
    > python 2.7, but could go to 3.x if that has better support for this.
    >
    > My program interacts with two different commercial libraries by using
    > ctypes to execute functions inside their DLLs. One library only
    > exists in a 32 bit version. The second library has both 64 and 32 bit
    > versions. The code was initially developed for a 32 bit platform, but
    > now I need to support a user on a 64 bit machine as well. I've
    > learned that ctypes can only interact with libraries which match the
    > version of python that is running --- 32 bit python can use 32 bit
    > DLLs, 64 bit python can use 64 bit DLLs, so when I'm running on a 64
    > bit machine with one 64 bit library and the other at 32 bits, I can't
    > use the 32 bit library.
    >
    > I believe that the solution is to have both win32 and x64 versions of
    > python installed, and then execute the win32 code inside of a new
    > process which uses the win32 version of Python. I'm already using
    > multiprocessing to run that code inside a new process because the two
    > libraries have conflicts and need to run in separate memory space. If
    > I could just specify that I want the new process to use a different
    > version of the interpreter I think I'd be all set, but I don't see a
    > way of doing that.


    You cannot use a Process() to do this, but you should be able to use Managers or
    the other tools in the multiprocessing packages for communicating between the
    processes.

    http://docs.python.org/library/multiprocessing#module-multiprocessing.managers

    There may be hidden assumptions in the code, though. I've never tried it myself.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Oct 4, 2010
    #2
    1. Advertising

  3. Scott Pigman

    Robert Kern Guest

    Re: Can multiprocessing.Process() use a different version of theinterpreter?

    On 10/4/10 11:39 AM, Scott Pigman wrote:

    > I believe that the solution is to have both win32 and x64 versions of
    > python installed, and then execute the win32 code inside of a new
    > process which uses the win32 version of Python. I'm already using
    > multiprocessing to run that code inside a new process because the two
    > libraries have conflicts and need to run in separate memory space. If
    > I could just specify that I want the new process to use a different
    > version of the interpreter I think I'd be all set, but I don't see a
    > way of doing that.


    My previous statement notwithstanding, there is a way to hack this. The
    windows-specific implementation of the class multiprocessing.forking.Popen uses
    a global variable, _python_exe, to determine the executable to use. Instead, you
    could subclass Popen to use whatever executable you like and subclass Process to
    use this Popen. See the source files process.py and forking.py in the
    multiprocessing package. It should be straightforward to attempt, but I don't
    know for sure that it will work.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Oct 4, 2010
    #3
  4. Scott Pigman

    Scott Pigman Guest

    On Oct 4, 1:34 pm, Robert Kern <> wrote:

    > My previous statement notwithstanding, there is a way to hack this. The
    > windows-specific implementation of the class multiprocessing.forking.Popen uses
    > a global variable, _python_exe, to determine the executable to use. Instead, you


    What I've been looking at since I first posted is that that
    _python.exe variable is built from sys.exec_prefix. So, I think that
    I can make it work by resetting sys.exec_prefix to the install dir of
    the 32 bit version before creating the new Process().
    Scott Pigman, Oct 4, 2010
    #4
  5. Scott Pigman

    Scott Pigman Guest

    On Oct 4, 1:41 pm, Scott Pigman <> wrote:

    > _python.exe variable is built from sys.exec_prefix.  So, I think that


    sorry, _python_exe.
    Scott Pigman, Oct 4, 2010
    #5
  6. Scott Pigman

    Robert Kern Guest

    Re: Can multiprocessing.Process() use a different version of theinterpreter?

    On 10/4/10 12:43 PM, Scott Pigman wrote:
    > On Oct 4, 1:41 pm, Scott Pigman<> wrote:
    >
    >> _python.exe variable is built from sys.exec_prefix. So, I think that

    >
    > sorry, _python_exe.


    Personally, I think that subclassing is cleaner, but if you must monkeypatch,
    then you should monkeypatch the multiprocessing.forking._python_exe variable,
    not sys.exec_prefix.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Oct 4, 2010
    #6
  7. Scott Pigman

    Scott Pigman Guest


    >
    > Personally, I think that subclassing is cleaner, but if you must monkeypatch,
    > then you should monkeypatch the multiprocessing.forking._python_exe variable,
    > not sys.exec_prefix.
    >
    > --
    > Robert Kern


    thanks, good point.
    Scott Pigman, Oct 4, 2010
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page