Inter-process communication, how?

[ecir.hana]

Hi,
let's say I have two scripts: one does some computations and the other
one is a graphical front end for launching the first one. And both run
in separate processes (front end runs and that it spawns a subprocess
with the computation). Now, if the computation has a result I would
like to display it in the front end. In another words, I would like to
pass some data from one process to another. How to do that? I'm
affraid I can't use a pipe since the computation could print out some
logging (if I understant pipes correctly).
Thanks!

 

[Dennis Lee Bieber]

Hi,
let's say I have two scripts: one does some computations and the other
one is a graphical front end for launching the first one. And both run
in separate processes (front end runs and that it spawns a subprocess
with the computation). Now, if the computation has a result I would
like to display it in the front end. In another words, I would like to
pass some data from one process to another. How to do that? I'm
affraid I can't use a pipe since the computation could print out some
logging (if I understant pipes correctly).
Thanks!

Have you perused and been perplexed by the "Is Python a Scripting
Language" thread? <G>

Question: Are you creating both scripts from scratch?
Yes? Then you can define whatever protocol is needed for your usage and
is available on your OS.

If it is a one-shot (spawn sub, wait, retrieve results) you could
generate a temporary file name in the parent, pass that name to the sub
when invoking it, wait for the sub to complete (giving you a status
code) and then read the result the sub has written to the file.

Or, you could have parent and sub both mmap the same "file",
essentially passing data in memory unless it becomes too large and pages
out to swap disk. You might even be able to do bidirectional and dynamic
updates (rather than waiting for the sub to exit)... Define, say, an
epoch count for each process -- these would be the first two words of
the mmap file

|p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|

periodically the parent would examine the value of s-epoch, and if it
has changed, read the s-data.. (both sides write the data first, then
update the epoch count). When the epoch stays the same and two
consecutive reads of the data match, you have stable data (so the reads
should occur more often than updates) and can process the data
transferred.

OR, you could have the parent open a TCP/IP socket as a server, and
pass the socket port number to the sub. The sub would then connect to
that port and write the data. For bidirectional you could pass the
parent port, and the sub's first action is to connect and pass a port
that it will be monitoring.

On a VMS system, the processes would connect to named "mailboxes"
and use QIO operations to pass data between them.

On an Amiga you'd use "message ports" (which operated somewhat
similar to VMS mailboxes except that mailboxes had an independent
existence, multiple processes can read or write to them -- message ports
were readable by the creating process, but could have messages sent from
anywhere; typically passing the message port [address of a linked list
of messages] for replies). Or a higher level message port: an ARexx
port.

On a Windows NT class system, the win32 extensions allow access to
Windows Named Pipes... Or maybe the Windows clipboard could be used...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[ecir.hana]

Have you perused and been perplexed by the "Is Python a Scripting
Language" thread? <G>

Oh, no!! Script as in "shell script".

Question: Are you creating both scripts from scratch?
Yes?

Yes.


Then you can define whatever protocol is needed for your usage and

is available on your OS.

If it is a one-shot (spawn sub, wait, retrieve results) you could
generate a temporary file name in the parent, pass that name to the sub
when invoking it, wait for the sub to complete (giving you a status
code) and then read the result the sub has written to the file.

Yes, it's once shot. But how do I pass "that name"?
From all the arguments of class Popen (http://docs.python.org/lib/
node529.html) perhaps I could use "env" only. Or am I wrong?
TCP/IP sounds good but isn't it a bit too heavy?
And what the other options goes I would prefer a cross platform
solution, if there is one.

PS: both with mmpam and temp file you probably meant that I should
hard code some as-weirdest-filename-as-possible two both programs but
what if I run the computation several times?

And thanks for reply!

Or, you could have parent and sub both mmap the same "file",
essentially passing data in memory unless it becomes too large and pages
out to swap disk. You might even be able to do bidirectional and dynamic
updates (rather than waiting for the sub to exit)... Define, say, an
epoch count for each process -- these would be the first two words of
the mmap file

|p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|

periodically the parent would examine the value of s-epoch, and if it
has changed, read the s-data.. (both sides write the data first, then
update the epoch count). When the epoch stays the same and two
consecutive reads of the data match, you have stable data (so the reads
should occur more often than updates) and can process the data
transferred.

OR, you could have the parent open a TCP/IP socket as a server, and
pass the socket port number to the sub. The sub would then connect to
that port and write the data. For bidirectional you could pass the
parent port, and the sub's first action is to connect and pass a port
that it will be monitoring.

On a VMS system, the processes would connect to named "mailboxes"
and use QIO operations to pass data between them.

On an Amiga you'd use "message ports" (which operated somewhat
similar to VMS mailboxes except that mailboxes had an independent
existence, multiple processes can read or write to them -- message ports
were readable by the creating process, but could have messages sent from
anywhere; typically passing the message port [address of a linked list
of messages] for replies). Or a higher level message port: an ARexx
port.

On a Windows NT class system, the win32 extensions allow access to
Windows Named Pipes... Or maybe the Windows clipboard could be used...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[John Machin]

Yes, it's once shot. But how do I pass "that name"?
From all the arguments of class Popen (http://docs.python.org/lib/
node529.html) perhaps I could use "env" only. Or am I wrong?

Yes. Consider this: If you were to run your calculation script from
the shell prompt [strongly recommended during testing], how would you
tell it the name of the file? Now look at the docs again.

PS: both with mmpam and temp file you probably meant that I should
hard code some as-weirdest-filename-as-possible two both programs but
what if I run the computation several times?

That's mmap, not mmpam. No, Dennis didn't mean that you should hard
code a filename. Have a look at the tempfile module.

And thanks for reply!

Or, you could have parent and sub both mmap the same "file",
essentially passing data in memory unless it becomes too large and pages
out to swap disk. You might even be able to do bidirectional and dynamic
updates (rather than waiting for the sub to exit)... Define, say, an
epoch count for each process -- these would be the first two words of
the mmap file

|p-epoch|s-epoch|n-words of p-data (fixed constant known to both)|n-words of s-data|

periodically the parent would examine the value of s-epoch, and if it
has changed, read the s-data.. (both sides write the data first, then
update the epoch count). When the epoch stays the same and two
consecutive reads of the data match, you have stable data (so the reads
should occur more often than updates) and can process the data
transferred.

OR, you could have the parent open a TCP/IP socket as a server, and
pass the socket port number to the sub. The sub would then connect to
that port and write the data. For bidirectional you could pass the
parent port, and the sub's first action is to connect and pass a port
that it will be monitoring.

On a VMS system, the processes would connect to named "mailboxes"
and use QIO operations to pass data between them.

On an Amiga you'd use "message ports" (which operated somewhat
similar to VMS mailboxes except that mailboxes had an independent
existence, multiple processes can read or write to them -- message ports
were readable by the creating process, but could have messages sent from
anywhere; typically passing the message port [address of a linked list
of messages] for replies). Or a higher level message port: an ARexx
port.

On a Windows NT class system, the win32 extensions allow access to
Windows Named Pipes... Or maybe the Windows clipboard could be used...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Dennis Lee Bieber]

Oh, no!! Script as in "shell script".

I only mentioned that thread as part of it devolved into a
discussion of IPC... <G> {I'll take the blame for that, having mentioned
Amiga's ARexx being able to "address" application native commands to any
program that exposed an ARexx port -- ARexx also sends any statement
that is not recognized by the interpreter to the last addressed port, so
one didn't have the hassle of explicitly managing the interface}

Yes, it's once shot. But how do I pass "that name"?
From all the arguments of class Popen (http://docs.python.org/lib/
node529.html) perhaps I could use "env" only. Or am I wrong?
TCP/IP sounds good but isn't it a bit too heavy?
And what the other options goes I would prefer a cross platform
solution, if there is one.

Read the details for subprocess.Popen() again...

"""
/args/ should be a string, or a sequence of program arguments. The
program to execute is normally the first item in the args sequence or
string, but can be explicitly set by using the executable argument.
"""

IOWs, passing it what you would enter on a command line

"subscript.py tempfilename"
["subscript.py", "tempfilename"]

should be sufficient.

PS: both with mmpam and temp file you probably meant that I should
hard code some as-weirdest-filename-as-possible two both programs but
what if I run the computation several times?

There is a module that can generate temporary file names, though for
this usage you could even do something to obtain the parent program
process ID along with a timestamp and create a file name from all that.
What are the odds that your "several times" would have the same clock
time?
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[ecir.hana]

Yes. Consider this: If you were to run your calculation script from
the shell prompt [strongly recommended during testing], how would you
tell it the name of the file? Now look at the docs again.

File arguments! Of course, totally forgot about them! Thanks a lot!

That's mmap, not mmpam. No, Dennis didn't mean that you should hard
code a filename. Have a look at the tempfile module.

Now I recall I read I somewhere that network communication is so much
faster than disk access (especially on the same machine, as steve
howell suggested) so instead of file names I probably should find out
which port is open to use.

 

[ecir.hana]

Read the details for subprocess.Popen() again...

"""
/args/ should be a string, or a sequence of program arguments. The
program to execute is normally the first item in the args sequence or
string, but can be explicitly set by using the executable argument.
"""

IOWs, passing it what you would enter on a command line

"subscript.py tempfilename"
["subscript.py", "tempfilename"]

should be sufficient.

There is a module that can generate temporary file names, though for
this usage you could even do something to obtain the parent program
process ID along with a timestamp and create a file name from all that.
What are the odds that your "several times" would have the same clock
time?

Quite small, I guess. However, perhaps I should better consider using
sockets.

Thanks!

ps: I really like how you format the paragraphs!

 

[ecir.hana]

Just for the record:

http://www.amk.ca/python/howto/sockets/

"Of the various forms of IPC (Inter Process Communication), sockets
are by far the most popular. On any given platform, there are likely
to be other forms of IPC that are faster, but for cross-platform
communication, sockets are about the only game in town."

 

[Nikita the Spider]

Hi,
let's say I have two scripts: one does some computations and the other
one is a graphical front end for launching the first one. And both run
in separate processes (front end runs and that it spawns a subprocess
with the computation). Now, if the computation has a result I would
like to display it in the front end. In another words, I would like to
pass some data from one process to another. How to do that? I'm
affraid I can't use a pipe since the computation could print out some
logging (if I understant pipes correctly).

Others have given you good suggestions; there's also this option which
may or may not be an appropriate tool for what you want to do:
http://NikitaTheSpider.com/python/shm/