Inter-process communication, how? Part 2

[ecir.hana]

Hello,

just to recap: last time I asked how to do an interprocess
communitation, between one Manager process (graphical beckend) and
some Worker processes.

I decided to go with sockets, thanks for replies, once more.

However, I would like to ask another thing: I would like to collect
everyting what the Workers print and display in Manager. Or, redirect
all Workers' stdout to stdio of Manager. If there was only one Worker
I could use a pipe, right? But if there are more than one Worker, what
to do? I found something called "named pipe" which seems rather
complicated. Then I thought I could somehow (how?) create a fake
(virtual) file object, redirect stdout of a Worket into it and from
there send the data to Manager via sockets. Please, what do you think?

Preferably, it should look like this:

--- Worker 1 ---
....some code...
print '123'

--- Manager ---
Worker 1: 123

--- Worker 2 ---
....some code...
print '456'

--- Manager ---
Worker 1: 123
Worker 2: 456

Thanks in advance!

 

[Dennis Lee Bieber]

just to recap: last time I asked how to do an interprocess
communitation, between one Manager process (graphical beckend) and
some Worker processes.

Never considered a graphical control process as a "backend"
before... Backend processing, to me, implies some sort of server without
a user interface.

However, I would like to ask another thing: I would like to collect
everyting what the Workers print and display in Manager. Or, redirect
all Workers' stdout to stdio of Manager. If there was only one Worker
I could use a pipe, right? But if there are more than one Worker, what
to do? I found something called "named pipe" which seems rather
complicated. Then I thought I could somehow (how?) create a fake
(virtual) file object, redirect stdout of a Worket into it and from
there send the data to Manager via sockets. Please, what do you think?

I'd forget about stdout as a data communication means... The parent
should probably set up a socket that accepts messages from any worker...
or create a "reply" socket for each worker, and pass the worker the port
on which the master expects to retrieve its output.

"Named pipes" are, I think, a M$ Windows creation (though I think
the Amiga supported disjoint pipes by using "run program >pipe:name" and
"program <pipe:name" instead of "program | program"
http://stason.org/TULARC/pc/amiga/faq/2-5-1-Using-PIPE-in-a-standard-AmigaShell-environment.html
-- "run program" being ~ "program &" in most UNIX-based shells)
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Guilherme Polo]

Hello,

just to recap: last time I asked how to do an interprocess
communitation, between one Manager process (graphical beckend) and
some Worker processes.

I decided to go with sockets, thanks for replies, once more.

However, I would like to ask another thing: I would like to collect
everyting what the Workers print and display in Manager. Or, redirect
all Workers' stdout to stdio of Manager. If there was only one Worker
I could use a pipe, right? But if there are more than one Worker, what
to do? I found something called "named pipe" which seems rather
complicated.

Named pipe is called FIFO, but they are not that complicated.

Then I thought I could somehow (how?) create a fake
(virtual) file object,

That is why it is called named pipe, because you will be using a
(especial) file in your filesystem to use as a pipe.

redirect stdout of a Worket into it and from
there send the data to Manager via sockets. Please, what do you think?

Preferably, it should look like this:

--- Worker 1 ---
...some code...
print '123'

--- Manager ---
Worker 1: 123

--- Worker 2 ---
...some code...
print '456'

--- Manager ---
Worker 1: 123
Worker 2: 456

Thanks in advance!

Manager would create and open the FIFO, Workers would open this same
FIFO. So workers write to FIFO and the manager reads from FIFO.

 

[ecir.hana]

Never considered a graphical control process as a "backend"
before... Backend processing, to me, implies some sort of server without
a user interface.


I'd forget about stdout as a data communication means... The parent
should probably set up a socket that accepts messages from any worker...
or create a "reply" socket for each worker, and pass the worker the port
on which the master expects to retrieve its output.

Ok, but how to redirect "print" statement into a socket?

 

[ecir.hana]

Named pipe is called FIFO, but they are not that complicated.


That is why it is called named pipe, because you will be using a
(especial) file in your filesystem to use as a pipe.










Manager would create and open the FIFO, Workers would open this same
FIFO. So workers write to FIFO and the manager reads from FIFO.

What I don't like about FIFO, is that on Unix they are persistent
files. So whatever happens to Manager they would stay there...
I was just wondering if there's another way of doing the above and if
not, I would probably go with FIFO. Thanks!

 

[Bjoern Schliessmann]

What I don't like about FIFO, is that on Unix they are persistent
files. So whatever happens to Manager they would stay there...

Nope. /tmp exists. Many distributions delete /tmp contents on
reboot.

I was just wondering if there's another way of doing the above and
if not, I would probably go with FIFO. Thanks!

Didn't you write you went with sockets?

Regards,


Björn

 

[Bjoern Schliessmann]

Ok, but how to redirect "print" statement into a socket?

Create an object that has the socket's send-or-what-it's-called
method as a member called "write" and bind this object to
sys.stdout. Alternatively, you can use the "print "text" >> object"
syntax (see language reference).

Regards,


Björn

 

[John Nagle]

Hello,

just to recap: last time I asked how to do an interprocess
communitation, between one Manager process (graphical beckend) and
some Worker processes.

I decided to go with sockets, thanks for replies, once more.

However, I would like to ask another thing: I would like to collect
everyting what the Workers print and display in Manager. Or, redirect
all Workers' stdout to stdio of Manager. If there was only one Worker
I could use a pipe, right? But if there are more than one Worker, what
to do? I found something called "named pipe" which seems rather
complicated. Then I thought I could somehow (how?) create a fake
(virtual) file object, redirect stdout of a Worket into it and from
there send the data to Manager via sockets. Please, what do you think?

Take a look at

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52296

which does this. I wouldn'd do it quite that way; read the notes
on the article for some criticism. But it's an approach that works.

One can get fancier. I wrote a system under QNX where each
real time process had its text output labeled, timestamped, and transmitted
using QNX interprocess communication to a lower priority logging process
on a different machine. Text output never blocked; if a queue
filled, "..." appeared in the log file. That's a special real-time situation.

John Nagle

 

[Hendrik van Rooyen]

What I don't like about FIFO, is that on Unix they are persistent
files. So whatever happens to Manager they would stay there...
I was just wondering if there's another way of doing the above and if
not, I would probably go with FIFO. Thanks!

The persistence bit is true, and can lead to puzzling behaviour when you
are still debugging, as the recipient can get a mix of old and new data
after a crash.

It is probably best to delete them and create them for every run, either
in a BASH script, or in the Python programme that first opens the pipe.

As for the alternative, on Linux, sockets seem to hang around for a while
after a run, until the OS deletes them, so that repetitive execution can give
you a socket in use error.

So you pays yer money and you takes yer pick.

- Hendrik

 

[Bjoern Schliessmann]

The persistence bit is true, and can lead to puzzling behaviour
when you are still debugging, as the recipient can get a mix of
old and new data after a crash.

It is probably best to delete them and create them for every run,
either in a BASH script, or in the Python programme that first
opens the pipe.

You could also drain the pipe in a script using

As for the alternative, on Linux, sockets seem to hang around for
a while after a run, until the OS deletes them, so that repetitive
execution can give you a socket in use error.

In my experience this only happens if you kill the program. If it
terminates normally it should close its socket so it is "freed"
immediately.

Grüße,


Björn

 

[Guilherme Polo]

You could also drain the pipe in a script using

To solve that "problem" with socket in use error, you should set the
SO_REUSEADDR socket option in the server before calling bind.

In my experience this only happens if you kill the program. If it
terminates normally it should close its socket so it is "freed"
immediately.

Supossing you run the client and server on the same machine, then when
the client finishes the connection normally, client final state is
TIME_WAIT (and this is not a bad thing actually, like some people
seems to think).

Why TIME_WAIT state exists: to allow old duplicates segments to expire
in the network, and, to implement TCP's full-duplex connection
termination reliability.

 

[makkalot]

Sunday 23 December 2007 Tarihinde 03:56:05 yazmıştı:

Hello,

just to recap: last time I asked how to do an interprocess
communitation, between one Manager process (graphical beckend) and
some Worker processes.

Why just dont use dbus , and then have a drink