Re: Pass data to a subprocess

Discussion in 'Python' started by Laszlo Nagy, Aug 1, 2012.

  1. Laszlo Nagy

    Laszlo Nagy Guest

    >
    > As I wrote "I found many nice things (Pipe, Manager and so on), but
    > actually even
    > this seems to work:" yes I did read the documentation.

    Sorry, I did not want be offensive.
    >
    > I was just surprised that it worked better than I expected even
    > without Pipes and Queues, but now I understand why..
    >
    > Anyway now I would like to be able to detach subprocesses to avoid the
    > nasty code reloading that I was talking about in another thread, but
    > things get more tricky, because I can't use queues and pipes to
    > communicate with a running process that it's noit my child, correct?
    >

    Yes, I think that is correct. Instead of detaching a child process, you
    can create independent processes and use other frameworks for IPC. For
    example, Pyro. It is not as effective as multiprocessing.Queue, but in
    return, you will have the option to run your service across multiple
    servers.

    The most effective IPC is usually through shared memory. But there is no
    OS independent standard Python module that can communicate over shared
    memory. Except multiprocessing of course, but AFAIK it can only be used
    to communicate between fork()-ed processes.
     
    Laszlo Nagy, Aug 1, 2012
    #1
    1. Advertising

  2. Laszlo Nagy

    Roy Smith Guest

    In article <>,
    Laszlo Nagy <> wrote:

    > Yes, I think that is correct. Instead of detaching a child process, you
    > can create independent processes and use other frameworks for IPC. For
    > example, Pyro. It is not as effective as multiprocessing.Queue, but in
    > return, you will have the option to run your service across multiple
    > servers.


    You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
    We've been using it in production for the better part of two years. At
    a 30,000 foot level, it's an implementation of queues over named pipes
    over TCP, but it takes care of a zillion little details for you.

    Setup is trivial, and there's clients for all sorts of languages. For a
    Python client, go with beanstalkc (pybeanstalk appears to be
    abandonware).
    >
    > The most effective IPC is usually through shared memory. But there is no
    > OS independent standard Python module that can communicate over shared
    > memory.


    It's true that shared memory is faster than serializing objects over a
    TCP connection. On the other hand, it's hard to imagine anything
    written in Python where you would notice the difference.
     
    Roy Smith, Aug 1, 2012
    #2
    1. Advertising

  3. Laszlo Nagy

    Laszlo Nagy Guest


    >> The most effective IPC is usually through shared memory. But there is no
    >> OS independent standard Python module that can communicate over shared
    >> memory.

    > It's true that shared memory is faster than serializing objects over a
    > TCP connection. On the other hand, it's hard to imagine anything
    > written in Python where you would notice the difference.

    Well, except in response times. ;-)

    The TCP stack likes to wait after you call send() on a socket. Yes, you
    can use setsockopt/TCP_NOWAIT, but my experience is that response times
    with TCP can be long, especially when you have to do many
    request-response pairs.

    It also depends on the protocol design - if you can reduce the number of
    request-response pairs then it helps a lot.
     
    Laszlo Nagy, Aug 1, 2012
    #3
  4. Laszlo Nagy

    Laszlo Nagy Guest

    On 2012-08-01 12:59, Roy Smith wrote:
    > In article <>,
    > Laszlo Nagy <> wrote:
    >
    >> Yes, I think that is correct. Instead of detaching a child process, you
    >> can create independent processes and use other frameworks for IPC. For
    >> example, Pyro. It is not as effective as multiprocessing.Queue, but in
    >> return, you will have the option to run your service across multiple
    >> servers.

    > You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
    > We've been using it in production for the better part of two years. At
    > a 30,000 foot level, it's an implementation of queues over named pipes
    > over TCP, but it takes care of a zillion little details for you.

    Looks very simple to use. Too bad that it doesn't work on Windows systems.
     
    Laszlo Nagy, Aug 1, 2012
    #4
  5. 2012/8/1 Roy Smith <>:
    > In article <>,
    > Laszlo Nagy <> wrote:
    >
    >> Yes, I think that is correct. Instead of detaching a child process, you
    >> can create independent processes and use other frameworks for IPC. For
    >> example, Pyro. It is not as effective as multiprocessing.Queue, but in
    >> return, you will have the option to run your service across multiple
    >> servers.

    >
    > You might want to look at beanstalk (http://kr.github.com/beanstalkd/).
    > We've been using it in production for the better part of two years. At
    > a 30,000 foot level, it's an implementation of queues over named pipes
    > over TCP, but it takes care of a zillion little details for you.
    >
    > Setup is trivial, and there's clients for all sorts of languages. For a
    > Python client, go with beanstalkc (pybeanstalk appears to be
    > abandonware).
    >>
    >> The most effective IPC is usually through shared memory. But there is no
    >> OS independent standard Python module that can communicate over shared
    >> memory.

    >
    > It's true that shared memory is faster than serializing objects over a
    > TCP connection. On the other hand, it's hard to imagine anything
    > written in Python where you would notice the difference.
    > --
    > http://mail.python.org/mailman/listinfo/python-list



    That does look nice and I would like to have something like that..
    But since I have to convince my boss of another external dependency I
    think it might be worth
    to try out zeromq instead, which can also do similar things and looks
    more powerful, what do you think?
     
    andrea crotti, Aug 1, 2012
    #5
  6. On 2012-08-01, Laszlo Nagy <> wrote:
    >>
    >> As I wrote "I found many nice things (Pipe, Manager and so on), but
    >> actually even
    >> this seems to work:" yes I did read the documentation.

    > Sorry, I did not want be offensive.
    >>
    >> I was just surprised that it worked better than I expected even
    >> without Pipes and Queues, but now I understand why..
    >>
    >> Anyway now I would like to be able to detach subprocesses to avoid the
    >> nasty code reloading that I was talking about in another thread, but
    >> things get more tricky, because I can't use queues and pipes to
    >> communicate with a running process that it's noit my child, correct?
    >>

    > Yes, I think that is correct.


    I don't understand why detaching a child process on Linux/Unix would
    make IPC stop working. Can somebody explain?

    --
    Grant Edwards grant.b.edwards Yow! My vaseline is
    at RUNNING...
    gmail.com
     
    Grant Edwards, Aug 1, 2012
    #6
  7. Laszlo Nagy

    Laszlo Nagy Guest


    >>> things get more tricky, because I can't use queues and pipes to
    >>> communicate with a running process that it's noit my child, correct?
    >>>

    >> Yes, I think that is correct.

    > I don't understand why detaching a child process on Linux/Unix would
    > make IPC stop working. Can somebody explain?
    >

    It is implemented with shared memory. I think (although I'm not 100%
    sure) that shared memory is created *and freed up* (shm_unlink() system
    call) by the parent process. It makes sense, because the child processes
    will surely die with the parent. If you detach a child process, then it
    won't be killed with its original parent. But the shared memory will be
    freed by the original parent process anyway. I suspect that the child
    that has mapped that shared memory segment will try to access a freed up
    resource, do a segfault or something similar.
     
    Laszlo Nagy, Aug 1, 2012
    #7
  8. Laszlo Nagy

    Laszlo Nagy Guest


    >>> Yes, I think that is correct.

    >> I don't understand why detaching a child process on Linux/Unix would
    >> make IPC stop working. Can somebody explain?
    >>

    > It is implemented with shared memory. I think (although I'm not 100%
    > sure) that shared memory is created *and freed up* (shm_unlink()
    > system call) by the parent process. It makes sense, because the child
    > processes will surely die with the parent. If you detach a child
    > process, then it won't be killed with its original parent. But the
    > shared memory will be freed by the original parent process anyway. I
    > suspect that the child that has mapped that shared memory segment will
    > try to access a freed up resource, do a segfault or something similar.

    So detaching the child process will not make IPC stop working. But
    exiting from the original parent process will. (And why else would you
    detach the child?)
     
    Laszlo Nagy, Aug 1, 2012
    #8
  9. 2012/8/1 Laszlo Nagy <>:
    >
    > So detaching the child process will not make IPC stop working. But exiting
    > from the original parent process will. (And why else would you detach the
    > child?)
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list



    Well it makes perfect sense if it stops working to me, so or
    - I use zeromq or something similar to communicate
    - I make every process independent without the need to further
    communicate with the parent..
     
    andrea crotti, Aug 1, 2012
    #9
  10. Laszlo Nagy

    Roy Smith Guest

    On Aug 1, 2012, at 9:25 AM, andrea crotti wrote:

    > [beanstalk] does look nice and I would like to have something like that..
    > But since I have to convince my boss of another external dependency I
    > think it might be worth
    > to try out zeromq instead, which can also do similar things and looks
    > more powerful, what do you think?


    I'm afraid I have no experience with zeromq, so I can't offer an opinion.

    --
    Roy Smith
     
    Roy Smith, Aug 1, 2012
    #10
  11. On 2012-08-01, Laszlo Nagy <> wrote:
    >
    >>>> things get more tricky, because I can't use queues and pipes to
    >>>> communicate with a running process that it's noit my child, correct?
    >>>>
    >>> Yes, I think that is correct.

    >> I don't understand why detaching a child process on Linux/Unix would
    >> make IPC stop working. Can somebody explain?

    >
    > It is implemented with shared memory. I think (although I'm not 100%
    > sure) that shared memory is created *and freed up* (shm_unlink() system
    > call) by the parent process. It makes sense, because the child processes
    > will surely die with the parent. If you detach a child process, then it
    > won't be killed with its original parent. But the shared memory will be
    > freed by the original parent process anyway. I suspect that the child
    > that has mapped that shared memory segment will try to access a freed up
    > resource, do a segfault or something similar.


    I still don't get it. shm_unlink() works the same way unlink() does.
    The resource itself doesn't cease to exist until all open file handles
    are closed. From the shm_unlink() man page on Linux:

    The operation of shm_unlink() is analogous to unlink(2): it
    removes a shared memory object name, and, once all processes
    have unmapped the object, de-allocates and destroys the
    contents of the associated memory region. After a successful
    shm_unlink(), attempts to shm_open() an object with the same
    name will fail (unless O_CREAT was specified, in which case a
    new, distinct object is created).

    Even if the parent calls shm_unlink(), the shared-memory resource will
    continue to exist (and be usable) until all processes that are holding
    open file handles unmap/close them. So not only will detached
    children not crash, they'll still be able to use the shared memory
    objects to talk to each other.

    --
    Grant Edwards grant.b.edwards Yow! Why is it that when
    at you DIE, you can't take
    gmail.com your HOME ENTERTAINMENT
    CENTER with you??
     
    Grant Edwards, Aug 1, 2012
    #11
  12. Laszlo Nagy

    Laszlo Nagy Guest


    > I still don't get it. shm_unlink() works the same way unlink() does.
    > The resource itself doesn't cease to exist until all open file handles
    > are closed. From the shm_unlink() man page on Linux:
    >
    > The operation of shm_unlink() is analogous to unlink(2): it
    > removes a shared memory object name, and, once all processes
    > have unmapped the object, de-allocates and destroys the
    > contents of the associated memory region. After a successful
    > shm_unlink(), attempts to shm_open() an object with the same
    > name will fail (unless O_CREAT was specified, in which case a
    > new, distinct object is created).
    >
    > Even if the parent calls shm_unlink(), the shared-memory resource will
    > continue to exist (and be usable) until all processes that are holding
    > open file handles unmap/close them. So not only will detached
    > children not crash, they'll still be able to use the shared memory
    > objects to talk to each other.
    >

    I stand corrected. It should still be examined, what kind shared memory
    is used under non-linux systems. System V on AIX? And what about
    Windows? So maybe the general answer is still no. But I guess that the
    OP wanted this to work on a specific system.

    Dear Andrea Crotti! Please try to detach two child processes, exit from
    the main process, and communicate over a multiprocessing queue. It will
    possibly work. Sorry for my bad advice.
     
    Laszlo Nagy, Aug 2, 2012
    #12
  13. On 2012-08-02, Laszlo Nagy <> wrote:
    >
    >> I still don't get it. shm_unlink() works the same way unlink() does.
    >> The resource itself doesn't cease to exist until all open file
    >> handles are closed. From the shm_unlink() man page on Linux:
    >>
    >> The operation of shm_unlink() is analogous to unlink(2): it
    >> removes a shared memory object name, and, once all processes
    >> have unmapped the object, de-allocates and destroys the
    >> contents of the associated memory region. After a successful
    >> shm_unlink(), attempts to shm_open() an object with the same
    >> name will fail (unless O_CREAT was specified, in which case a
    >> new, distinct object is created).
    >>
    >> Even if the parent calls shm_unlink(), the shared-memory resource
    >> will continue to exist (and be usable) until all processes that are
    >> holding open file handles unmap/close them. So not only will
    >> detached children not crash, they'll still be able to use the shared
    >> memory objects to talk to each other.


    Note that when I say the detached children will still be able to talk
    to each other using shared memory after the parent calls shm_unlink()
    and exit(), I'm talking about the general case -- not specifically
    about the multiprocessing module. There may be something else going on
    with the multiprocessing module.

    > I stand corrected. It should still be examined, what kind shared
    > memory is used under non-linux systems. System V on AIX? And what
    > about Windows? So maybe the general answer is still no. But I guess
    > that the OP wanted this to work on a specific system.
    >
    > Dear Andrea Crotti! Please try to detach two child processes, exit
    > from the main process, and communicate over a multiprocessing queue.
    > It will possibly work. Sorry for my bad advice.


    I'm not claiming it will work, since I don't know how the IPC in the
    multiprocessing module works. It may indeed break when a child
    process is detatched (which I'm assuming means being removed from the
    process group and/or detached from the controlling tty).

    But, I'm not aware of any underlying Unix IPC mechanism that breaks
    when a child is detached, so I was curious about what would cause
    multiprocessing's IPC to break.

    --
    Grant Edwards grant.b.edwards Yow! I didn't order any
    at WOO-WOO ... Maybe a YUBBA
    gmail.com ... But no WOO-WOO!
     
    Grant Edwards, Aug 2, 2012
    #13
  14. On Thu, 02 Aug 2012 08:10:59 +0200, Laszlo Nagy <>
    declaimed the following in gmane.comp.python.general:


    > I stand corrected. It should still be examined, what kind shared memory
    > is used under non-linux systems. System V on AIX? And what about
    > Windows? So maybe the general answer is still no. But I guess that the
    > OP wanted this to work on a specific system.
    >

    From an ancient MSDN disk (goes back to VB6):

    """
    Shared Memory
    The Win32 API uses a special case of file mapping to provide shared
    memory access between processes. If you specify the system swap file
    when creating a file-mapping object, the file-mapping object is treated
    as a shared memory block. Other processes can access the same block of
    memory by opening the same file-mapping object (see File Mapping).
    Because shared memory is implemented with file mapping, it supports
    security access attributes and can operate only between processes
    running on the same computer.

    For information about shared memory, see "General Library" in the
    "Windows Base Services" section of the Microsoft Platform SDK.
    """

    There is also the MFC CSharedFile class.

    .NET prior to Framework v4 did not implement mapped files.

    I believe Python exposes this (the Win32 aspect) via mmap.

    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
     
    Dennis Lee Bieber, Aug 2, 2012
    #14
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. andrea crotti

    Pass data to a subprocess

    andrea crotti, Jul 31, 2012, in forum: Python
    Replies:
    0
    Views:
    185
    andrea crotti
    Jul 31, 2012
  2. andrea crotti

    Re: Pass data to a subprocess

    andrea crotti, Jul 31, 2012, in forum: Python
    Replies:
    0
    Views:
    163
    andrea crotti
    Jul 31, 2012
  3. Laszlo Nagy

    Re: Pass data to a subprocess

    Laszlo Nagy, Jul 31, 2012, in forum: Python
    Replies:
    0
    Views:
    180
    Laszlo Nagy
    Jul 31, 2012
  4. andrea crotti

    Re: Pass data to a subprocess

    andrea crotti, Jul 31, 2012, in forum: Python
    Replies:
    0
    Views:
    152
    andrea crotti
    Jul 31, 2012
  5. andrea crotti

    Re: Pass data to a subprocess

    andrea crotti, Aug 1, 2012, in forum: Python
    Replies:
    0
    Views:
    175
    andrea crotti
    Aug 1, 2012
Loading...

Share This Page