simple and fast platform independent IPC

[News123]

Hi,

I wondered what IPC library might be best simplest for following task?

I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )

these scripts want to send short messages to each other ( mostly
integers, max a few bytes, short string), which would be enqueued in
message queues of the receiving process.

example:

'dad' wants to tell 'mom': 'cook'
'dog' wants to tell 'dad' : 'feedme'
'mom' wants to tell 'dad' : 'cookyourself'

the receiver dos not necesarily have to know who sent the message

a message shall be dropped silently if the receiving process is not running

a message shall be reliably delivered if the receiving process is up


xmlrpc seems to be a little heavy for such tasks.

signals don't allow to exchange data

a shared memory message queue would probably a good solution, but
python's Multiprocessing.Queue seems to require a common parent process



thanks a lot for any ideas / suggestions



N



N

 

[Gabriel Genellina]

I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent
process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )

these scripts want to send short messages to each other ( mostly
integers, max a few bytes, short string), which would be enqueued in
message queues of the receiving process.

example:

'dad' wants to tell 'mom': 'cook'
'dog' wants to tell 'dad' : 'feedme'
'mom' wants to tell 'dad' : 'cookyourself'

Try using a named pipe between each pair of processes (os.mkfifo + open on
Linux, open(r"\\.\pipe\desired_name", ...) on Windows)

 

[Vinay Sajip]

Hi,

I wondered what IPC library might be best simplest for following task?

I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )

these scripts want to send short messages to each other ( mostly
integers, max a few bytes, short string), which would be enqueued in
message queues of the receiving process.

example:

'dad' wants to tell 'mom': 'cook'
'dog' wants to tell 'dad' : 'feedme'
'mom' wants to tell 'dad' : 'cookyourself'

the receiver dos not necesarily have to know who sent the message

a message shall be dropped silently if the receiving process is not running

a message shall be reliably delivered if the receiving process is up

xmlrpc seems to be a little heavy for such tasks.

signals don't allow to exchange data

a shared memory message queue would probably a good solution, but
python's Multiprocessing.Queue  seems to require a common parent process

thanks a lot for any ideas / suggestions

N

N

Gabriel's suggestion is very good; if you need something which is a
little more like RPC but still quite lightweight, consider Pyro
(http://pyro.sourceforge.net/)

Regards,

Vinay Sajip

 

[Joan Miller]

Gabriel's suggestion is very good; if you need something which is a
little more like RPC but still quite lightweight, consider Pyro
(http://pyro.sourceforge.net/)

Regards,

Vinay Sajip

I've read that Pyro is not safe. Anyway, you have in mind that respect
to speed:

shared memory > named pipes > Unix domain socket > TCP socket

I don't sure about if the message queues would be faster that Unix
domain sockets

Another thing. Using shared memory would be as to use a single thread
but using message queues would be as multiple-threading.

 

[News123]

Hi Gabriel,

I'll look at it.
I wasn't aware about named pipes for windows.

bye


N

 

[Tim Golden]

....

[Vinay Sajip]

Gabriel's suggestion is very good; if you need something which is a
little more like RPC but still quite lightweight, consider Pyro
(http://pyro.sourceforge.net/)

[[email protected]]
I've read that Pyro is not safe.

That's a fairly broad thing to say. I've read lots
of things. What does "is not safe" mean, in any case?
I assume you've got a valid concern in mind which is
worth passing on to a would-be user, but what exactly
is it? FWIW I've used Pyro on and off over the years
without any problems. Certainly my computer's never
blown up as a result of using it.

Obviously Pyro is Python-only so interaction with non-Python
code would be problematic. But the OP only mentions Python
scripts so hopefully that wouldn't be an issue...

Anyway, you have in mind that respect to speed:

shared memory> named pipes> Unix domain socket> TCP socket

True, but the OP didn't mention speed; rather simplicity. Not
saying it isn't a consideration but premature optimisation and
all that...

Another thing. Using shared memory would be as to use a single thread
but using message queues would be as multiple-threading.

And therefore...?

I think you need to make your points more clearly.

TJG

 

[Eden Kirin]

Hi,

I wondered what IPC library might be best simplest for following task?

Consider using Thrift (http://incubator.apache.org/thrift/). It is
multiplatform multilanguage RPC and IPC solution. I implemented it in
couple of my projects and it works seamlessly.

 

[Joan Miller]

I wondered what IPC library might be best simplest for following task?

...

xmlrpc seems to be a little heavy for such tasks.
signals don't allow to exchange data
a shared memory message queue would probably a good solution, but
python's Multiprocessing.Queue  seems to require a common parent process

[Vinay Sajip]

Gabriel's suggestion is very good; if you need something which is a
little more like RPC but still quite lightweight, consider Pyro
(http://pyro.sourceforge.net/)

[[email protected]]

I've read that Pyro is not safe.

That's a fairly broad thing to say. I've read lots
of things. What does "is not safe" mean, in any case?
I assume you've got a valid concern in mind which is
worth passing on to a would-be user, but what exactly
is it? FWIW I've used Pyro on and off over the years
without any problems. Certainly my computer's never
blown up as a result of using it.

From its own page:
"Pyro has never been truly designed to provide a secure communication
mechanism, nor has it had a security review or -test by a security
expert."
http://pyro.sourceforge.net/features.html

 

[Paul Rubin]

I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )

If they are running on the same host with no untrusted local users, you
can use unix-domain sockets instead of TCP sockets and then the server
should be unreachable from the internet. Then you're less exposed to
possible security issues with libraries like Pyro. Personally I've just
used SocketServer/SimpleHTTPServer on the listening side and simplejson
for serialization, but that was for low-rent, low-performance
applications. If you want something faster, zeromq (www.zeromq.org)
looks interesting.

 

[Terry Reedy]

"Pyro has never been truly designed to provide a secure communication
mechanism, nor has it had a security review or -test by a security
expert."
http://pyro.sourceforge.net/features.html

For communication between processes on one machine, that hardly seems to
be a worry. If it were, I would expect that sending encrypted strings
would substantially improve things.

That aside, I would wonder whether you could use a master process with a
gui to haphazardly launch subprocess, so as to avail oneself of
multiprocessing.Queue.

Terry Jan Reedy

 

[News123]

True, but the OP didn't mention speed; rather simplicity. Not
saying it isn't a consideration but premature optimisation and
all that...

Yes true. I'm looking for something simple, platform agnostic. Lateron I
can always improve performance.

Perhaps I'll stick initially with xmlrpc, as it is quite simple, though
a little heavy.
I just have to see how to deal with servers which are not up, no more
up, being restarted.


N

 

[Paul Rubin]

Perhaps I'll stick initially with xmlrpc, as it is quite simple,
though a little heavy. I just have to see how to deal with servers
which are not up, no more up, being restarted.

Something wrong wtih nagios?

 

[News123]

Hi Terry,

That aside, I would wonder whether you could use a master process with a
gui to haphazardly launch subprocess, so as to avail oneself of
multiprocessing.Queue.

T

This is also an option I'm looking.

insted of the python scripts, thet users would normally click at I had
to implement small sripts, that just communicate with the master
process, which would then launch the application.

 

[bobicanprogram]

Hi,

I wondered what IPC library might be best simplest for following task?

I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )

these scripts want to send short messages to each other ( mostly
integers, max a few bytes, short string), which would be enqueued in
message queues of the receiving process.

example:

'dad' wants to tell 'mom': 'cook'
'dog' wants to tell 'dad' : 'feedme'
'mom' wants to tell 'dad' : 'cookyourself'

the receiver dos not necesarily have to know who sent the message

a message shall be dropped silently if the receiving process is not running

a message shall be reliably delivered if the receiving process is up

xmlrpc seems to be a little heavy for such tasks.

signals don't allow to exchange data

a shared memory message queue would probably a good solution, but
python's Multiprocessing.Queue seems to require a common parent process

thanks a lot for any ideas / suggestions

N

N

Have a look at the SIMPL toolkit (http://www.icanprogram.com/simpl or
http://www.icanprogram.com/06py/main.html). It should do everything
you describe at least on the Linux end of the spectrum. Under the
hood SIMPL uses a shared memory scheme for local message exchange.
SIMPL would be especially effective if you want to move your modules
at some future point onto different nodes. SIMPL-Python can work
seamlessly on heterogeneous networks provided at least one node is
Linux.

bob