Re: Multiprocessing, shared memory vs. pickled copies

Discussion in 'Python' started by Miki Tebeka, Apr 10, 2011.

  1. Miki Tebeka

    Miki Tebeka Guest

    > Now, I don't know that I actually HAVE to pass my neural network and
    > input data as copies -- they're both READ-ONLY objects for the
    > duration of an evaluate function (which can go on for quite a while).

    One option in that case is to use "fork" (if you're on a *nix machine).
    See http://pythonwise.blogspot.com/2009/04/pmap.html for example ;)

    HTH
    --
    Miki Tebeka <>
    http://pythonwise.blogspot.com
     
    Miki Tebeka, Apr 10, 2011
    #1
    1. Advertising

  2. Miki Tebeka

    John Nagle Guest

    On 4/10/2011 9:11 AM, Miki Tebeka wrote:
    >> Now, I don't know that I actually HAVE to pass my neural network and
    >> input data as copies -- they're both READ-ONLY objects for the
    >> duration of an evaluate function (which can go on for quite a while).

    > One option in that case is to use "fork" (if you're on a *nix machine).
    > See http://pythonwise.blogspot.com/2009/04/pmap.html for example ;)


    Unless you have a performance problem, don't bother with shared
    memory.

    If you have a performance problem, Python is probably the wrong
    tool for the job anyway.

    John Nagle
     
    John Nagle, Apr 10, 2011
    #2
    1. Advertising

  3. Miki Tebeka

    sturlamolden Guest

    On 10 apr, 18:27, John Nagle <> wrote:

    >     Unless you have a performance problem, don't bother with shared
    > memory.
    >
    >     If you have a performance problem, Python is probably the wrong
    > tool for the job anyway.


    Then why does Python have a multiprocessing module?

    In my opinion, if Python has a multiprocessing module in the standard
    library, it should also be possible to use it with NumPy.

    Sturla
     
    sturlamolden, Apr 10, 2011
    #3
  4. Miki Tebeka

    John Nagle Guest

    On 4/10/2011 3:29 PM, sturlamolden wrote:
    > On 10 apr, 18:27, John Nagle<> wrote:
    >
    >> Unless you have a performance problem, don't bother with shared
    >> memory.
    >>
    >> If you have a performance problem, Python is probably the wrong
    >> tool for the job anyway.

    >
    > Then why does Python have a multiprocessing module?


    Because nobody can fix the Global Interpreter Lock problem in CPython.

    The multiprocessing module is a hack to get around the fact
    that Python threads don't run concurrently, and thus, threaded
    programs don't effectively use multi-core CPUs.

    John Nagle
     
    John Nagle, Apr 11, 2011
    #4
  5. Miki Tebeka

    sturlamolden Guest

    On 11 apr, 09:21, John Nagle <> wrote:

    >      Because nobody can fix the Global Interpreter Lock problem in CPython.
    >
    >      The multiprocessing module is a hack to get around the fact
    > that Python threads don't run concurrently, and thus, threaded
    > programs don't effectively use multi-core CPUs.


    Then why not let NumPy use shared memory in this case? It is as simple
    as this:

    import numpy as np
    import sharedmem as sm
    private_array = np.zeros((10,10))
    shared_array = sm.zeros((10,10))

    I.e. the code is identical (more or less).

    Also, do you think shared memory has other uses besides parallel
    computing, such as IPC? It is e.g. an easy way to share data between
    Python and an external program. Do you e.g. need to share (not send)
    large amounts of data between Python and Java or Matlab?

    The difference here from multiprocessing.Array is that IPC with
    multiprocessing.Queue will actually work, because I am not using
    anonymous shared memory. Instead of inherting handles, the segment has
    a name in the file system, which means it can be memory mapped from
    anywhere.

    I posted the code to the NumPy mailing list yesterday.

    Sturla
     
    sturlamolden, Apr 11, 2011
    #5
  6. Miki Tebeka

    sturlamolden Guest

    On 11 apr, 21:11, sturlamolden <> wrote:

    > import numpy as np
    > import sharedmem as sm
    > private_array = np.zeros((10,10))
    > shared_array  = sm.zeros((10,10))


    I am also using this to implement synchronization primitives (barrier,
    lock, semaphore, event) that can be sent over an instance of
    multiprocessing.Queue. The synchronization primitives in
    multiprocessing cannot be communicated this way.

    Sturla
     
    sturlamolden, Apr 11, 2011
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Felix
    Replies:
    1
    Views:
    876
    Robert Kern
    Oct 8, 2009
  2. Valery
    Replies:
    9
    Views:
    1,487
    Klauss
    Jan 7, 2010
  3. Kevin Ar18
    Replies:
    0
    Views:
    221
    Kevin Ar18
    Jul 27, 2010
  4. Kevin Ar18
    Replies:
    0
    Views:
    230
    Kevin Ar18
    Jul 27, 2010
  5. John Ladasky
    Replies:
    21
    Views:
    2,255
    sturlamolden
    Apr 10, 2011
Loading...

Share This Page