Inter-process communication, how?

Discussion in 'Python' started by ecir.hana@gmail.com, Dec 16, 2007.

  1. Guest

    Hi,
    let's say I have two scripts: one does some computations and the other
    one is a graphical front end for launching the first one. And both run
    in separate processes (front end runs and that it spawns a subprocess
    with the computation). Now, if the computation has a result I would
    like to display it in the front end. In another words, I would like to
    pass some data from one process to another. How to do that? I'm
    affraid I can't use a pipe since the computation could print out some
    logging (if I understant pipes correctly).
    Thanks!
     
    , Dec 16, 2007
    #1
    1. Advertising

  2. On Sat, 15 Dec 2007 16:44:34 -0800 (PST), declaimed
    the following in comp.lang.python:

    > Hi,
    > let's say I have two scripts: one does some computations and the other
    > one is a graphical front end for launching the first one. And both run
    > in separate processes (front end runs and that it spawns a subprocess
    > with the computation). Now, if the computation has a result I would
    > like to display it in the front end. In another words, I would like to
    > pass some data from one process to another. How to do that? I'm
    > affraid I can't use a pipe since the computation could print out some
    > logging (if I understant pipes correctly).
    > Thanks!


    Have you perused and been perplexed by the "Is Python a Scripting
    Language" thread? <G>

    Question: Are you creating both scripts from scratch?
    Yes? Then you can define whatever protocol is needed for your usage and
    is available on your OS.

    If it is a one-shot (spawn sub, wait, retrieve results) you could
    generate a temporary file name in the parent, pass that name to the sub
    when invoking it, wait for the sub to complete (giving you a status
    code) and then read the result the sub has written to the file.

    Or, you could have parent and sub both mmap the same "file",
    essentially passing data in memory unless it becomes too large and pages
    out to swap disk. You might even be able to do bidirectional and dynamic
    updates (rather than waiting for the sub to exit)... Define, say, an
    epoch count for each process -- these would be the first two words of
    the mmap file

    |p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|

    periodically the parent would examine the value of s-epoch, and if it
    has changed, read the s-data.. (both sides write the data first, then
    update the epoch count). When the epoch stays the same and two
    consecutive reads of the data match, you have stable data (so the reads
    should occur more often than updates) and can process the data
    transferred.

    OR, you could have the parent open a TCP/IP socket as a server, and
    pass the socket port number to the sub. The sub would then connect to
    that port and write the data. For bidirectional you could pass the
    parent port, and the sub's first action is to connect and pass a port
    that it will be monitoring.

    On a VMS system, the processes would connect to named "mailboxes"
    and use QIO operations to pass data between them.

    On an Amiga you'd use "message ports" (which operated somewhat
    similar to VMS mailboxes except that mailboxes had an independent
    existence, multiple processes can read or write to them -- message ports
    were readable by the creating process, but could have messages sent from
    anywhere; typically passing the message port [address of a linked list
    of messages] for replies). Or a higher level message port: an ARexx
    port.

    On a Windows NT class system, the win32 extensions allow access to
    Windows Named Pipes... Or maybe the Windows clipboard could be used...
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Dec 16, 2007
    #2
    1. Advertising

  3. Guest

    On Dec 16, 2:42 am, Dennis Lee Bieber <> wrote:
    >
    > Have you perused and been perplexed by the "Is Python a Scripting
    > Language" thread? <G>


    Oh, no!! Script as in "shell script".


    >
    > Question: Are you creating both scripts from scratch?
    > Yes?


    Yes.


    Then you can define whatever protocol is needed for your usage and
    > is available on your OS.
    >
    > If it is a one-shot (spawn sub, wait, retrieve results) you could
    > generate a temporary file name in the parent, pass that name to the sub
    > when invoking it, wait for the sub to complete (giving you a status
    > code) and then read the result the sub has written to the file.


    Yes, it's once shot. But how do I pass "that name"?
    From all the arguments of class Popen (http://docs.python.org/lib/
    node529.html) perhaps I could use "env" only. Or am I wrong?
    TCP/IP sounds good but isn't it a bit too heavy?
    And what the other options goes I would prefer a cross platform
    solution, if there is one.

    PS: both with mmpam and temp file you probably meant that I should
    hard code some as-weirdest-filename-as-possible two both programs but
    what if I run the computation several times?

    And thanks for reply!


    >
    > Or, you could have parent and sub both mmap the same "file",
    > essentially passing data in memory unless it becomes too large and pages
    > out to swap disk. You might even be able to do bidirectional and dynamic
    > updates (rather than waiting for the sub to exit)... Define, say, an
    > epoch count for each process -- these would be the first two words of
    > the mmap file
    >
    > |p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|
    >
    > periodically the parent would examine the value of s-epoch, and if it
    > has changed, read the s-data.. (both sides write the data first, then
    > update the epoch count). When the epoch stays the same and two
    > consecutive reads of the data match, you have stable data (so the reads
    > should occur more often than updates) and can process the data
    > transferred.
    >
    > OR, you could have the parent open a TCP/IP socket as a server, and
    > pass the socket port number to the sub. The sub would then connect to
    > that port and write the data. For bidirectional you could pass the
    > parent port, and the sub's first action is to connect and pass a port
    > that it will be monitoring.
    >
    > On a VMS system, the processes would connect to named "mailboxes"
    > and use QIO operations to pass data between them.
    >
    > On an Amiga you'd use "message ports" (which operated somewhat
    > similar to VMS mailboxes except that mailboxes had an independent
    > existence, multiple processes can read or write to them -- message ports
    > were readable by the creating process, but could have messages sent from
    > anywhere; typically passing the message port [address of a linked list
    > of messages] for replies). Or a higher level message port: an ARexx
    > port.
    >
    > On a Windows NT class system, the win32 extensions allow access to
    > Windows Named Pipes... Or maybe the Windows clipboard could be used...
    > --
    > Wulfraed Dennis Lee Bieber KD6MOG
    >
    > HTTP://wlfraed.home.netcom.com/
    > (Bestiaria Support Staff: )
    > HTTP://www.bestiaria.com/
     
    , Dec 16, 2007
    #3
  4. John Machin Guest

    On Dec 16, 2:34 pm, wrote:
    > > If it is a one-shot (spawn sub, wait, retrieve results) you could
    > > generate a temporary file name in the parent, pass that name to the sub
    > > when invoking it, wait for the sub to complete (giving you a status
    > > code) and then read the result the sub has written to the file.

    >
    > Yes, it's once shot. But how do I pass "that name"?
    > From all the arguments of class Popen (http://docs.python.org/lib/
    > node529.html) perhaps I could use "env" only. Or am I wrong?


    Yes. Consider this: If you were to run your calculation script from
    the shell prompt [strongly recommended during testing], how would you
    tell it the name of the file? Now look at the docs again.
    >
    > PS: both with mmpam and temp file you probably meant that I should
    > hard code some as-weirdest-filename-as-possible two both programs but
    > what if I run the computation several times?


    That's mmap, not mmpam. No, Dennis didn't mean that you should hard
    code a filename. Have a look at the tempfile module.


    >
    > And thanks for reply!
    >
    >
    >
    > > Or, you could have parent and sub both mmap the same "file",
    > > essentially passing data in memory unless it becomes too large and pages
    > > out to swap disk. You might even be able to do bidirectional and dynamic
    > > updates (rather than waiting for the sub to exit)... Define, say, an
    > > epoch count for each process -- these would be the first two words of
    > > the mmap file

    >
    > > |p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|

    >
    > > periodically the parent would examine the value of s-epoch, and if it
    > > has changed, read the s-data.. (both sides write the data first, then
    > > update the epoch count). When the epoch stays the same and two
    > > consecutive reads of the data match, you have stable data (so the reads
    > > should occur more often than updates) and can process the data
    > > transferred.

    >
    > > OR, you could have the parent open a TCP/IP socket as a server, and
    > > pass the socket port number to the sub. The sub would then connect to
    > > that port and write the data. For bidirectional you could pass the
    > > parent port, and the sub's first action is to connect and pass a port
    > > that it will be monitoring.

    >
    > > On a VMS system, the processes would connect to named "mailboxes"
    > > and use QIO operations to pass data between them.

    >
    > > On an Amiga you'd use "message ports" (which operated somewhat
    > > similar to VMS mailboxes except that mailboxes had an independent
    > > existence, multiple processes can read or write to them -- message ports
    > > were readable by the creating process, but could have messages sent from
    > > anywhere; typically passing the message port [address of a linked list
    > > of messages] for replies). Or a higher level message port: an ARexx
    > > port.

    >
    > > On a Windows NT class system, the win32 extensions allow access to
    > > Windows Named Pipes... Or maybe the Windows clipboard could be used...
    > > --
    > > Wulfraed Dennis Lee Bieber KD6MOG
    > >
    > > HTTP://wlfraed.home.netcom.com/
    > > (Bestiaria Support Staff: )
    > > HTTP://www.bestiaria.com/
     
    John Machin, Dec 16, 2007
    #4
  5. On Sat, 15 Dec 2007 19:34:37 -0800 (PST), declaimed
    the following in comp.lang.python:

    > On Dec 16, 2:42 am, Dennis Lee Bieber <> wrote:
    > >
    > > Have you perused and been perplexed by the "Is Python a Scripting
    > > Language" thread? <G>

    >
    > Oh, no!! Script as in "shell script".


    I only mentioned that thread as part of it devolved into a
    discussion of IPC... <G> {I'll take the blame for that, having mentioned
    Amiga's ARexx being able to "address" application native commands to any
    program that exposed an ARexx port -- ARexx also sends any statement
    that is not recognized by the interpreter to the last addressed port, so
    one didn't have the hassle of explicitly managing the interface}

    > Yes, it's once shot. But how do I pass "that name"?
    > From all the arguments of class Popen (http://docs.python.org/lib/
    > node529.html) perhaps I could use "env" only. Or am I wrong?
    > TCP/IP sounds good but isn't it a bit too heavy?
    > And what the other options goes I would prefer a cross platform
    > solution, if there is one.
    >

    Read the details for subprocess.Popen() again...

    """
    /args/ should be a string, or a sequence of program arguments. The
    program to execute is normally the first item in the args sequence or
    string, but can be explicitly set by using the executable argument.
    """

    IOWs, passing it what you would enter on a command line

    "subscript.py tempfilename"
    ["subscript.py", "tempfilename"]

    should be sufficient.

    > PS: both with mmpam and temp file you probably meant that I should
    > hard code some as-weirdest-filename-as-possible two both programs but
    > what if I run the computation several times?
    >

    There is a module that can generate temporary file names, though for
    this usage you could even do something to obtain the parent program
    process ID along with a timestamp and create a file name from all that.
    What are the odds that your "several times" would have the same clock
    time?
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Dec 16, 2007
    #5
  6. Guest

    On Dec 16, 5:24 am, John Machin <> wrote:
    >
    > Yes. Consider this: If you were to run your calculation script from
    > the shell prompt [strongly recommended during testing], how would you
    > tell it the name of the file? Now look at the docs again.
    >


    File arguments! Of course, totally forgot about them! Thanks a lot!

    >
    >
    > > PS: both with mmpam and temp file you probably meant that I should
    > > hard code some as-weirdest-filename-as-possible two both programs but
    > > what if I run the computation several times?

    >
    > That's mmap, not mmpam. No, Dennis didn't mean that you should hard
    > code a filename. Have a look at the tempfile module.


    Now I recall I read I somewhere that network communication is so much
    faster than disk access (especially on the same machine, as steve
    howell suggested) so instead of file names I probably should find out
    which port is open to use.
     
    , Dec 16, 2007
    #6
  7. Guest

    On Dec 16, 6:38 am, Dennis Lee Bieber <> wrote:
    >
    >
    > Read the details for subprocess.Popen() again...
    >
    > """
    > /args/ should be a string, or a sequence of program arguments. The
    > program to execute is normally the first item in the args sequence or
    > string, but can be explicitly set by using the executable argument.
    > """
    >
    > IOWs, passing it what you would enter on a command line
    >
    > "subscript.py tempfilename"
    > ["subscript.py", "tempfilename"]
    >
    > should be sufficient.
    >


    >
    > There is a module that can generate temporary file names, though for
    > this usage you could even do something to obtain the parent program
    > process ID along with a timestamp and create a file name from all that.
    > What are the odds that your "several times" would have the same clock
    > time?
    >


    Quite small, I guess. However, perhaps I should better consider using
    sockets.

    Thanks!

    ps: I really like how you format the paragraphs! :)
     
    , Dec 16, 2007
    #7
  8. Guest

    Just for the record:

    http://www.amk.ca/python/howto/sockets/

    "Of the various forms of IPC (Inter Process Communication), sockets
    are by far the most popular. On any given platform, there are likely
    to be other forms of IPC that are faster, but for cross-platform
    communication, sockets are about the only game in town."
     
    , Dec 16, 2007
    #8
  9. In article
    <>,
    wrote:

    > Hi,
    > let's say I have two scripts: one does some computations and the other
    > one is a graphical front end for launching the first one. And both run
    > in separate processes (front end runs and that it spawns a subprocess
    > with the computation). Now, if the computation has a result I would
    > like to display it in the front end. In another words, I would like to
    > pass some data from one process to another. How to do that? I'm
    > affraid I can't use a pipe since the computation could print out some
    > logging (if I understant pipes correctly).


    Others have given you good suggestions; there's also this option which
    may or may not be an appropriate tool for what you want to do:
    http://NikitaTheSpider.com/python/shm/

    --
    Philip
    http://NikitaTheSpider.com/
    Whole-site HTML validation, link checking and more
     
    Nikita the Spider, Dec 22, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Knute Johnson
    Replies:
    0
    Views:
    839
    Knute Johnson
    Jun 27, 2003
  2. Jon A. Cruz
    Replies:
    0
    Views:
    750
    Jon A. Cruz
    Jun 28, 2003
  3. Sudsy
    Replies:
    0
    Views:
    981
    Sudsy
    Jun 28, 2003
  4. Frank D. Greco
    Replies:
    1
    Views:
    2,222
    Keeger
    Jun 30, 2003
  5. Hugo Ferreira
    Replies:
    0
    Views:
    826
    Hugo Ferreira
    Dec 4, 2006
Loading...

Share This Page