Approaches of interprocess communication

[exhuma.twn]

Hi all,

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.

Let me list the ones I know of:

* Sockets
Advantage: Supported per se in nearly every programming language
without even the need to install additional packages
Disadvantage: Lot's of code to write, and it's kind of silly to
communicate via TCP/IP if the processes run on the same machine.

* Webservices
Advantage: Relatively easy to use, can work across different
languages
Disadvantage: Even more overhead on the TCP/IP side that simple
sockets, as really bulky SOAP messages need to be passed around.

* CORBA -- similar to webservices but more complicated to code.

* Shared memory
I don't know much about this subject.

Supposing both processes are written in Python, is there any other way
to achieve this? To me, shared memory sound the most suited approach.
But as said, I am still fuzzy in this area. Where can I find more
information on this subject?

 

[Gabriel Genellina]

Hi all,

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.

Let me list the ones I know of:

* Sockets
Advantage: Supported per se in nearly every programming language
without even the need to install additional packages
Disadvantage: Lot's of code to write, and it's kind of silly to
communicate via TCP/IP if the processes run on the same machine.

Not so much code, really.
(And I would expect that making a connection to "localhost" actually does
*not* go down up to the network card hardware layer, but I don't know for
real if this is the case or not).

* Webservices
Advantage: Relatively easy to use, can work across different
languages
Disadvantage: Even more overhead on the TCP/IP side that simple
sockets, as really bulky SOAP messages need to be passed around.

You could use XMLRPC, wich is a lot simpler (but less powerful).

* CORBA -- similar to webservices but more complicated to code.

I would stay away as far as I could.

* Shared memory
I don't know much about this subject.

You forget the most basic one, stdio redirection. Easy, available on
almost any language, but limited to just a pair of processes.
You can add queues and messages.

Supposing both processes are written in Python, is there any other way
to achieve this? To me, shared memory sound the most suited approach.
But as said, I am still fuzzy in this area. Where can I find more
information on this subject?

Pyro appears to be a good alternative (altough I've never used it yet).

 

[Daniel Nogradi]

Supposing you have two separate processes running on the same box,

what approach would you suggest to communicate between those two
processes.

Let me list the ones I know of:

* Sockets
Advantage: Supported per se in nearly every programming language
without even the need to install additional packages
Disadvantage: Lot's of code to write, and it's kind of silly to
communicate via TCP/IP if the processes run on the same machine.

* Webservices
Advantage: Relatively easy to use, can work across different
languages
Disadvantage: Even more overhead on the TCP/IP side that simple
sockets, as really bulky SOAP messages need to be passed around.

* CORBA -- similar to webservices but more complicated to code.

* Shared memory
I don't know much about this subject.

Supposing both processes are written in Python, is there any other way
to achieve this? To me, shared memory sound the most suited approach.
But as said, I am still fuzzy in this area. Where can I find more
information on this subject?

Hi, if your requirements are sufficiently light then pylinda might be
an easy-to-use solution:

http://www-users.cs.york.ac.uk/~aw/pylinda/

A simple example is here:

http://www-users.cs.york.ac.uk/~aw/pylinda/beginner.html

HTH,
Daniel

 

[Ben Finney]

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.

Let me list the ones I know of:

* Sockets
Advantage: Supported per se in nearly every programming language
without even the need to install additional packages

This would be my choice. But first, set up a well-defined *protocol*
(preferably based on text commands) for the two processes to use for
communication; don't have each of them being intricately aware of each
others' implementation.

Disadvantage: Lot's of code to write, and it's kind of silly to
communicate via TCP/IP if the processes run on the same machine.

You can cut down on the amount of code by using the standard library
"cmd" module to handle a command interface, hooking the stdin and
stdout of the commandline handler to the socket.

If you're already thinking about cooperating processes, you should
make them network-neutral anyway from the start.

Here's what _The Art of Unix Programming_ has to say on the topic of
text protocols:

<URL:http://www.catb.org/~esr/writings/taoup/html/ch05s01.html>

and IPC tactics for peer processes:

<URL:http://www.catb.org/~esr/writings/taoup/html/ch07s07.html>

 

[Ben Finney]

(And I would expect that making a connection to "localhost" actually
does *not* go down up to the network card hardware layer, but I
don't know for real if this is the case or not).

It damned well better. That's the entire point of the loopback
interface: to get all the network layer code involved, but not to talk
on a physical network interface.

If a programmer decides on behalf of the user that "localhost" should
be treated specially, that programmer is making an error.

 

[Duncan Grisby]

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.
[...]
* Webservices
Advantage: Relatively easy to use, can work across different
languages
Disadvantage: Even more overhead on the TCP/IP side that simple
sockets, as really bulky SOAP messages need to be passed around.

* CORBA -- similar to webservices but more complicated to code.

Lots of people say that, but I don't think it's true. Obviously as the
maintainer of a CORBA implementation I'm biased, but take a look at
some examples of code that implement SOAP clients and servers and
compare it to similar CORBA code. Especially in Python, the SOAP code
tends to be incredibly verbose and complex, and the CORBA code really
small and simple.

My recommendation would be that for simple communications where
performance isn't important use XML-RPC; for more complex cases where
performance is a bit more important but you don't need anything except
Python, use Pyro; for cases where performance is particularly
important and/or you need cross-language communications, use CORBA.

Cheers,

Duncan.

 

[exhuma.twn]

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.
[...]

* Webservices
Advantage: Relatively easy to use, can work across different
languages
Disadvantage: Even more overhead on the TCP/IP side that simple
sockets, as really bulky SOAP messages need to be passed around.

* CORBA -- similar to webservices but more complicated to code.

Lots of people say that, but I don't think it's true. Obviously as the
maintainer of a CORBA implementation I'm biased, but take a look at
some examples of code that implement SOAP clients and servers and
compare it to similar CORBA code. Especially in Python, the SOAP code
tends to be incredibly verbose and complex, and the CORBA code really
small and simple.

My recommendation would be that for simple communications where
performance isn't important use XML-RPC; for more complex cases where
performance is a bit more important but you don't need anything except
Python, use Pyro; for cases where performance is particularly
important and/or you need cross-language communications, use CORBA.

Maybe this line of mine was a bit too condensed I fully agree with
you on what you say about CORBA. It's just that for most people IDL
looks a bit out of place. Especially because it resembles C. But once
you actually wrote a few projects using CORBA, you actually begin to
see it's elegance

I find Webservices "easier" as you can (although it's not
recommendable) leave out the WSDL part. And some implementations
actually generate the WSDL for you. But it's true, comparing WSDL and
IDL I'd rather write some IDL than WSDL

But, returning back to topic, I first want to thank you all for your
input. I appreciate all the ideas. Especially the idea of using the
"cmd" module for sockets. That sound's very intriguing and I will very
likely play around with that. But also PYRO seems quite useful.

About "Linda": Am I right that it looks very similar to "JavaSpaces"?
If yes, are there any funcdamental differences between those two?

 

[Diez B. Roggisch]

Maybe this line of mine was a bit too condensed I fully agree with

you on what you say about CORBA. It's just that for most people IDL
looks a bit out of place. Especially because it resembles C. But once
you actually wrote a few projects using CORBA, you actually begin to
see it's elegance

It looks out of the place in comparison to what exactly? A WSDL-document?
You certainly me laugh hard on that....

I find Webservices "easier" as you can (although it's not
recommendable) leave out the WSDL part. And some implementations
actually generate the WSDL for you.

You can't leave WSDL out of SOAP, you can leave it out of XMLRPC. Which is
nice and easy, but lacks any contract whatsoever. Nothing I as a Pythoneer
care to much about, but some people bother.

And generating the WSDL works from what exactly? ah, Java-code for example,
using java2wsdl. Which has a striking resemblance to.... C. Funny....

But it's true, comparing WSDL and
IDL I'd rather write some IDL than WSDL

So - you say that yourself, but still you previously claimed IDL looks
strange to the eye?

Diez

 

[Paul Boddie]

You can't leave WSDL out of SOAP

Yes you can, since they're two different things. What you probably
meant was that you can't leave WSDL out of "big architecture", W3C
standards-intensive Web services. Of course, RPC-style SOAP without
the contracts imposed by WSDL may remove some of the attractions for
some people (who really should consider CORBA, anyway), but then
there's always document-oriented SOAP, although if you don't want the
baggage associated with routing and other things, plain XML messaging
would be easier. And there's always XMPP if you want stuff like
routing and a standard that isn't undergoing apparently continuous
change.

Paul

 

[Diez B. Roggisch]

Yes you can, since they're two different things. What you probably
meant was that you can't leave WSDL out of "big architecture", W3C
standards-intensive Web services. Of course, RPC-style SOAP without
the contracts imposed by WSDL may remove some of the attractions for
some people (who really should consider CORBA, anyway), but then
there's always document-oriented SOAP, although if you don't want the
baggage associated with routing and other things, plain XML messaging
would be easier. And there's always XMPP if you want stuff like
routing and a standard that isn't undergoing apparently continuous
change.

Didn't know that. Yet I presume it is pretty awful to manually decompose and
compose the method invocations and parameter sets.

I've got no idea what document-orientied SOAP means either. Is that just
pushing XML-files over a protocol?

Diez

 

[Paul Boddie]

[XMPP, XML messaging]

Didn't know that. Yet I presume it is pretty awful to manually decompose and
compose the method invocations and parameter sets.

It depends on how well you like working with XML, I suppose.

I've got no idea what document-orientied SOAP means either. Is that just
pushing XML-files over a protocol?

As I understand it, yes, although you still have to put the payload
inside the usual SOAP boilerplate.

Paul

 

[Steve Holden]

It damned well better. That's the entire point of the loopback
interface: to get all the network layer code involved, but not to talk
on a physical network interface.

If a programmer decides on behalf of the user that "localhost" should
be treated specially, that programmer is making an error.

Inter-process TCP/IP communication between two processes on the same
host invariably uses the loopback interface (network 127.0.0.0).
According to standards, all addresses in that network space refer to the
local host, though 127.0.0.1 is conventionally used.

The transmit driver for the loopback interface receives a datagram from
the local network layer and immediately announces its reception back to
the local network layer.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

 

[Roman Yakovenko]

Hi ,
We have a project where I need to read files store
them in database in the backend.We have done this in
python.Now we decided to use Ajax technique for user
interface.For that we found that GWT is one of the
best toolkits.Now I got a doubt can I interface GWT
with python.

http://pyjamas.pyworks.org/ is the way to go.

 

[Daniel Nogradi]

About "Linda": Am I right that it looks very similar to "JavaSpaces"?

If yes, are there any funcdamental differences between those two?

Yes, they are both linda implementations, but I have no idea what so
ever how they compare. A local java expert maybe?

 

[Steve Holden]

Hi ,
We have a project where I need to read files store
them in database in the backend.We have done this in
python.Now we decided to use Ajax technique for user
interface.For that we found that GWT is one of the
best toolkits.Now I got a doubt can I interface GWT
with python.
Thanks ,
Shadab.

Send instant messages to your online friends http://uk.messenger.yahoo.com

Please note that you should not create a new conversation on a newsgroup
by editing a reply to an unrelated post - your comments are now threaded
along wiht "Approaches of Interprocess Communication", making them
unnecessarily hard to find and somewhat confusingly positioned.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

 

[Steve Holden]

Hi ,
We have a project where I need to read files store
them in database in the backend.We have done this in
python.Now we decided to use Ajax technique for user
interface.For that we found that GWT is one of the
best toolkits.Now I got a doubt can I interface GWT
with python.
Thanks ,
Shadab.

Send instant messages to your online friends http://uk.messenger.yahoo.com

Please note that you should not create a new conversation on a newsgroup
by editing a reply to an unrelated post - your comments are now threaded
along wiht "Approaches of Interprocess Communication", making them
unnecessarily hard to find and somewhat confusingly positioned.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

 

[Dennis Lee Bieber]

You forget the most basic one, stdio redirection. Easy, available on
almost any language, but limited to just a pair of processes.
You can add queues and messages.

Named pipes on those OSs that support them (applicable if both
processes are on the same host). Even 20 year-old AmigaOS had a form of
named pipe (more a form of virtual file <G>).

mmap allows a form of shared memory, I believe, but doesn't really
support some means of safe locking. Probably useful if only one process
writes, and multiple processes are just reading (things like a satellite
telemetry decoder filling a block of fixed status data in a continuous
process, and other processes might be, say, a logger that runs at timed
intervals capturing whatever current status is in the blocks along with
a time-stamp, some control console wherein only selected status words
are being monitored and updating a screen at some other interval...)
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Bjoern Schliessmann]

* Sockets
Advantage: Supported per se in nearly every programming
language without even the need to install additional packages
Disadvantage: Lot's of code to write,

Who's Lot?

No, seriously. Why would you think that it's much to write? It can,
especially using Python, be as easy as using a file.

and it's kind of silly to communicate via TCP/IP if the
processes run on the same machine.

Why?

- Do you worry about CPU cycles, memory, ...?
- Who forces you to use TCP?

Supposing both processes are written in Python, is there any other
way to achieve this?

It depends on what those processes need to share. What you need can
range from a simple unix socket connection to a full-fledged SQL
database server.

Regards,


Björn

 

[Goldfish]

Hi all,

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.

Spring Python makes it easy to get processes talking to each other.
You can write your code like you are talking locally, then when its
time to separate it either into another thread, another python
process, or on another node, it is just a reconfiguration issue.

http://springpython.python-hosting.com/wiki/DistributedRemoting

 

[Nikita the Spider]

Hi all,

Supposing you have two separate processes running on the same box,
what approach would you suggest to communicate between those two
processes.

Hi exhuma,
That would depend on what data I was exchanging between the processes.
For instance, if process A spawns work process B and wants to be able
monitor B's progress, a message-based protocol might be kind of chatty.
In this situation shared memory is probably a better fit because B can
write it's progress to a chunk of shared memory and A can read that at
its leisure. OTOH if the conversation is more event-driven, then a
messaging protocol makes good sense.

FYI there's a Python module (with sample code) for using shared memory
on most *nix systems here:
http://NikitaTheSpider.com/python/shm/

HTH