suggestion on a complicated inter-process communication

[Way]

Hello friends,

I have a little messy situation on IPC. Please if you can, give me
some suggestion on how to implement. Thanks a lot!

-> denotes create


MainProcess -> Process1 -> Process3 (from os.system)
|
-> Process2 (from os.system) -> Process4 (from
os.system) ->Process5

I would like to make the communication between Process1 and Process5.
Process1 needs Process5's output to provide argument to generate
Process3, and in turn Process5 needs to wait Process3 finished.

Thank you very much if you can give a hint.

 

[Aaron Brady]

Hello friends,

I have a little messy situation on IPC. Please if you can, give me
some suggestion on how to implement. Thanks a lot!

-> denotes create

MainProcess -> Process1 -> Process3 (from os.system)
                   |
                    -> Process2 (from os.system) -> Process4 (from
os.system) ->Process5

I would like to make the communication between Process1 and Process5.
Process1 needs Process5's output to provide argument to generate
Process3, and in turn Process5 needs to wait Process3 finished.

Thank you very much if you can give a hint.

The 'mmap' module can help with getting the data from 5 to 1. It
requires creating a file. If it's a small amount of data, any old
file will do. You may need a socket in order to wait for Process3 to
join, or write a small '.pyd' or '.so' file that gives you access to
your system's synchronization object.

 

[Jonathan Gardner]

Hello friends,

I have a little messy situation on IPC. Please if you can, give me
some suggestion on how to implement. Thanks a lot!

-> denotes create

MainProcess -> Process1 -> Process3 (from os.system)
                   |
                    -> Process2 (from os.system) -> Process4 (from
os.system) ->Process5

I would like to make the communication between Process1 and Process5.
Process1 needs Process5's output to provide argument to generate
Process3, and in turn Process5 needs to wait Process3 finished.

Thank you very much if you can give a hint.

Abstraction should resolve this. What I mean is that Process2
shouldn't be defined in terms of what it actually does, but what it
appears to do. If you look at each process and think only what its
child processes do and what its parent process expects it to do, then
your job should get a lot simpler. Process1 expects Process2 to
deliver a set of parameters to spawn Process3, and then it will wait
until Process3 terminates.

Looking at it this way, questions come to mind: Why can't Process2 run
Process3 itself? It's unusual to have one process tell another process
what to do when it can simply do it itself.

By the way, I've never seen a time when this kind of design is
necessary. There are other ways around your problem than spawning a
bunch of processes. Each process should really be independent of all
the other processes, doing only a very well-specified task in a well-
specified way. The only time I could think of doing something like
this is when you're getting a webserver setup and Process5 needs to
communicate with Process3 to render the page or something like that.
But even in that case, each process is really independently defined
and implemented.

 

[Way]

Thanks a lot for reply. I understand it is abnormal to implement such
IPC, while it is worthy for my application.

Well, my process3 and 4 are from an outside application, which both
need License Check and os.system to involk.

From my experience, if Process5 involks Process3, such License Check
can be very very slow (even slower than its real run).

That is why I need to separate Process3 and 4 onto different processes
to proceed.

 

[Way]

Thanks a lot for the reply. I am not familiar with multi-process in
Python. I am now using something like:
A_prog is an os.system to involk Process3
B_prog is an os.system to involk Process4
-------------------------------------------------------
In Main Process:
Process1 = Popen(["A_prog"], stdin=PIPE, stdout=PIPE)
Process2 = Popen(["B_prog"], stdin=PIPE, stdout=PIPE)

cmd = Process2.stdout.readlines()
Process1.stdin.write(cmd)
if Process1.poll():
Process2.stdin.write("trig from Process1")

-----------------------------------------
In Process5 (another python program):
import sys
sys.stdout.write("process1 argument")
while 1:
if sys.stdin.readline().strip() == "trig from Process1":
break
------------------------------------------

However, in this case, Process5's stdout cannot be passed to
MainProcess for Process1, since it has not finished (waiting Process.
1/3 finish).

I am now using Fifos (an external file) to inter-communicate Process1
and 5. But at first run, it compliants "not file found".

Is there any possible solution to make it work? Thank you very much!

 

[Paul Boddie]

Thanks a lot for reply. I understand it is abnormal to implement such
IPC, while it is worthy for my application.

Well, my process3 and 4 are from an outside application, which both
need License Check and os.system to involk.

Sounds delightful!

Anyway, looking at your diagram (edited to fit)...

Main
| \-> P1 -> P3 (from os.system)
\-> P2 (from os.system) -> P4 (from os.system) ->P5

....if P1, P2 and P4 are able to propagate input and output, then this
is just a matter of having something in the main process which
monitors the input from P5 (via P4 and P2) and which relays the input
to P3 (via P1). This could be as simple as the following:

to_p1.write(from_p2.read())

If this isn't sophisticated enough, because you only want to read some
details from P2, you could either try and read a predetermined amount
from P2, or you could write a communications handler which monitors
the input from P2 using a select.poll object. If the main process is
supposed to be doing other things, then it's quite likely that you'll
have to do something involving either select.poll or threads, anyway.

Paul

 

[Paul Boddie]

Thanks a lot for the reply. I am not familiar with multi-process in
Python. I am now using something like:
A_prog is an os.system to involk Process3
B_prog is an os.system to involk Process4
-------------------------------------------------------
In Main Process:
Process1 = Popen(["A_prog"], stdin=PIPE, stdout=PIPE)
Process2 = Popen(["B_prog"], stdin=PIPE, stdout=PIPE)

cmd = Process2.stdout.readlines()

Careful here: this may want to read until process 2 has closed its
stream, which may only occur on exit, normally.

Process1.stdin.write(cmd)
if Process1.poll():
  Process2.stdin.write("trig from Process1")

Careful here, too: this output may not be sent immediately, due to
buffering issues.

while 1:
    if sys.stdin.readline().strip() == "trig from Process1":
            break

And this test may not be satisfied, since the main process may still
be waiting for output from process 5 (via process 2), not reaching the
point where it writes a response.

Paul

 

[norseman]

Hello friends,

I have a little messy situation on IPC. Please if you can, give me
some suggestion on how to implement. Thanks a lot!

-> denotes create


MainProcess -> Process1 -> Process3 (from os.system)
|
-> Process2 (from os.system) -> Process4 (from
os.system) ->Process5

I would like to make the communication between Process1 and Process5.
Process1 needs Process5's output to provide argument to generate
Process3, and in turn Process5 needs to wait Process3 finished.

Thank you very much if you can give a hint.

=====================================

My first reaction is to use named pipes.

Process1 -> P2 -> P4 -> Process5 >>NamedPipe (P5 outputs to NamedPipe)
Process1 waits for NamedPipe to be made and starts creating Process3
When Process3 is finished it can set an OS level environmental or
create a dummy file as a signal to Process5 to finish.
P1 becomes orchestrator, NamedPipe is courier, file is smoke signal.

This method works when IPC is not a good choice. (Like independent child
processes run across the net. A specific file in a specific location
effectively creates a specific signal. Whether or not contents are used
or even existing is programmer choice.)

Hope this helps.

Steve

 

[Aaron Brady]

Um, that's the limit of what I'm familiar with, I'm afraid. I'd have
to experiment.

Thanks a lot for the reply. I am not familiar with multi-process in
Python. I am now using something like: snip
However, in this case, Process5's stdout cannot be passed to
MainProcess for Process1, since it has not finished (waiting Process.
1/3 finish).

I am now using Fifos (an external file) to inter-communicate Process1
and 5. But at first run, it compliants "not file found".

Is there any possible solution to make it work? Thank you very much!

snip

 

[Lawrence D'Oliveiro]

-> denotes create


MainProcess -> Process1 -> Process3 (from os.system)
|
-> Process2 (from os.system) -> Process4 (from os.system) ->Process5

If MainProcess were to create two pairs of pipes with os.pipe, will they
successfully be passed down to all the child processes? If so, set the input
end of one pair so Process1 will read from it, and the output end of the
other pair so Process3 will write to it. Process5, which presumably is under
your control, can write to the output end of the first pair, and read from
the input end of the second pair.

 

[norseman]

Um, that's the limit of what I'm familiar with, I'm afraid. I'd have
to experiment.

If you are using a Named Pipe for P1's input, you won't have P5's stdout
to worry about. Besides, P1 should know the Named Pipe P5 is using
without Main telling P1. Give them their own "Open_Channel_D" to
communicate via. (The Man from U.N.C.L.E. -- probably before your time.)


Won't find it until P5 starts dumping into it OR P1 creates it for P5 to
use. Either way - P1 simply needs to poll it if/while there and for P5
going ByeBye to know when to stop polling. If P5 is gone, once the pipe
is empty, there will be no more data. If both (or either) P1, P5 are
constantly running they will need a "I'm done" signal for one to let the
other know it's going to sleep until needed. Last item passed is
something the sending program will NEVER generate on its own.
"Close_Channel_D"