Approaches of interprocess communication

Discussion in 'Python' started by exhuma.twn, Feb 16, 2007.

  1. exhuma.twn

    exhuma.twn Guest

    Hi all,

    Supposing you have two separate processes running on the same box,
    what approach would you suggest to communicate between those two
    processes.

    Let me list the ones I know of:

    * Sockets
    Advantage: Supported per se in nearly every programming language
    without even the need to install additional packages
    Disadvantage: Lot's of code to write, and it's kind of silly to
    communicate via TCP/IP if the processes run on the same machine.

    * Webservices
    Advantage: Relatively easy to use, can work across different
    languages
    Disadvantage: Even more overhead on the TCP/IP side that simple
    sockets, as really bulky SOAP messages need to be passed around.

    * CORBA -- similar to webservices but more complicated to code.

    * Shared memory
    I don't know much about this subject.

    Supposing both processes are written in Python, is there any other way
    to achieve this? To me, shared memory sound the most suited approach.
    But as said, I am still fuzzy in this area. Where can I find more
    information on this subject?
    exhuma.twn, Feb 16, 2007
    #1
    1. Advertising

  2. En Fri, 16 Feb 2007 07:11:36 -0300, exhuma.twn <> escribió:

    > Hi all,
    >
    > Supposing you have two separate processes running on the same box,
    > what approach would you suggest to communicate between those two
    > processes.
    >
    > Let me list the ones I know of:
    >
    > * Sockets
    > Advantage: Supported per se in nearly every programming language
    > without even the need to install additional packages
    > Disadvantage: Lot's of code to write, and it's kind of silly to
    > communicate via TCP/IP if the processes run on the same machine.


    Not so much code, really.
    (And I would expect that making a connection to "localhost" actually does
    *not* go down up to the network card hardware layer, but I don't know for
    real if this is the case or not).

    > * Webservices
    > Advantage: Relatively easy to use, can work across different
    > languages
    > Disadvantage: Even more overhead on the TCP/IP side that simple
    > sockets, as really bulky SOAP messages need to be passed around.


    You could use XMLRPC, wich is a lot simpler (but less powerful).

    > * CORBA -- similar to webservices but more complicated to code.


    I would stay away as far as I could.

    > * Shared memory
    > I don't know much about this subject.


    You forget the most basic one, stdio redirection. Easy, available on
    almost any language, but limited to just a pair of processes.
    You can add queues and messages.

    > Supposing both processes are written in Python, is there any other way
    > to achieve this? To me, shared memory sound the most suited approach.
    > But as said, I am still fuzzy in this area. Where can I find more
    > information on this subject?


    Pyro appears to be a good alternative (altough I've never used it yet).

    --
    Gabriel Genellina
    Gabriel Genellina, Feb 16, 2007
    #2
    1. Advertising

  3. > Supposing you have two separate processes running on the same box,
    > what approach would you suggest to communicate between those two
    > processes.
    >
    > Let me list the ones I know of:
    >
    > * Sockets
    > Advantage: Supported per se in nearly every programming language
    > without even the need to install additional packages
    > Disadvantage: Lot's of code to write, and it's kind of silly to
    > communicate via TCP/IP if the processes run on the same machine.
    >
    > * Webservices
    > Advantage: Relatively easy to use, can work across different
    > languages
    > Disadvantage: Even more overhead on the TCP/IP side that simple
    > sockets, as really bulky SOAP messages need to be passed around.
    >
    > * CORBA -- similar to webservices but more complicated to code.
    >
    > * Shared memory
    > I don't know much about this subject.
    >
    > Supposing both processes are written in Python, is there any other way
    > to achieve this? To me, shared memory sound the most suited approach.
    > But as said, I am still fuzzy in this area. Where can I find more
    > information on this subject?


    Hi, if your requirements are sufficiently light then pylinda might be
    an easy-to-use solution:

    http://www-users.cs.york.ac.uk/~aw/pylinda/

    A simple example is here:

    http://www-users.cs.york.ac.uk/~aw/pylinda/beginner.html

    HTH,
    Daniel
    Daniel Nogradi, Feb 16, 2007
    #3
  4. exhuma.twn

    Ben Finney Guest

    "exhuma.twn" <> writes:

    > Supposing you have two separate processes running on the same box,
    > what approach would you suggest to communicate between those two
    > processes.
    >
    > Let me list the ones I know of:
    >
    > * Sockets
    > Advantage: Supported per se in nearly every programming language
    > without even the need to install additional packages


    This would be my choice. But first, set up a well-defined *protocol*
    (preferably based on text commands) for the two processes to use for
    communication; don't have each of them being intricately aware of each
    others' implementation.

    > Disadvantage: Lot's of code to write, and it's kind of silly to
    > communicate via TCP/IP if the processes run on the same machine.


    You can cut down on the amount of code by using the standard library
    "cmd" module to handle a command interface, hooking the stdin and
    stdout of the commandline handler to the socket.

    If you're already thinking about cooperating processes, you should
    make them network-neutral anyway from the start.

    Here's what _The Art of Unix Programming_ has to say on the topic of
    text protocols:

    <URL:http://www.catb.org/~esr/writings/taoup/html/ch05s01.html>

    and IPC tactics for peer processes:

    <URL:http://www.catb.org/~esr/writings/taoup/html/ch07s07.html>

    --
    \ "There are only two ways to live your life. One is as though |
    `\ nothing is a miracle. The other is as if everything is." -- |
    _o__) Albert Einstein |
    Ben Finney
    Ben Finney, Feb 16, 2007
    #4
  5. exhuma.twn

    Ben Finney Guest

    "Gabriel Genellina" <> writes:

    > (And I would expect that making a connection to "localhost" actually
    > does *not* go down up to the network card hardware layer, but I
    > don't know for real if this is the case or not).


    It damned well better. That's the entire point of the loopback
    interface: to get all the network layer code involved, but not to talk
    on a physical network interface.

    If a programmer decides on behalf of the user that "localhost" should
    be treated specially, that programmer is making an error.

    --
    \ "If you can't beat them, arrange to have them beaten." -- |
    `\ George Carlin |
    _o__) |
    Ben Finney
    Ben Finney, Feb 16, 2007
    #5
  6. In article <>,
    exhuma.twn <> wrote:

    >Supposing you have two separate processes running on the same box,
    >what approach would you suggest to communicate between those two
    >processes.


    [...]
    >* Webservices
    > Advantage: Relatively easy to use, can work across different
    >languages
    > Disadvantage: Even more overhead on the TCP/IP side that simple
    >sockets, as really bulky SOAP messages need to be passed around.
    >
    >* CORBA -- similar to webservices but more complicated to code.


    Lots of people say that, but I don't think it's true. Obviously as the
    maintainer of a CORBA implementation I'm biased, but take a look at
    some examples of code that implement SOAP clients and servers and
    compare it to similar CORBA code. Especially in Python, the SOAP code
    tends to be incredibly verbose and complex, and the CORBA code really
    small and simple.

    My recommendation would be that for simple communications where
    performance isn't important use XML-RPC; for more complex cases where
    performance is a bit more important but you don't need anything except
    Python, use Pyro; for cases where performance is particularly
    important and/or you need cross-language communications, use CORBA.

    Cheers,

    Duncan.

    --
    -- Duncan Grisby --
    -- --
    -- http://www.grisby.org --

    --
    Posted via NewsDemon.com - Premium Uncensored Newsgroup Service
    ------->>>>>>http://www.NewsDem
    Duncan Grisby, Feb 16, 2007
    #6
  7. exhuma.twn

    exhuma.twn Guest

    On Feb 16, 1:33 pm, Duncan Grisby <> wrote:
    > In article <>,
    >
    > exhuma.twn <> wrote:
    > >Supposing you have two separate processes running on the same box,
    > >what approach would you suggest to communicate between those two
    > >processes.

    >
    > [...]
    >
    > >* Webservices
    > > Advantage: Relatively easy to use, can work across different
    > >languages
    > > Disadvantage: Even more overhead on the TCP/IP side that simple
    > >sockets, as really bulky SOAP messages need to be passed around.

    >
    > >* CORBA -- similar to webservices but more complicated to code.

    >
    > Lots of people say that, but I don't think it's true. Obviously as the
    > maintainer of a CORBA implementation I'm biased, but take a look at
    > some examples of code that implement SOAP clients and servers and
    > compare it to similar CORBA code. Especially in Python, the SOAP code
    > tends to be incredibly verbose and complex, and the CORBA code really
    > small and simple.
    >
    > My recommendation would be that for simple communications where
    > performance isn't important use XML-RPC; for more complex cases where
    > performance is a bit more important but you don't need anything except
    > Python, use Pyro; for cases where performance is particularly
    > important and/or you need cross-language communications, use CORBA.


    Maybe this line of mine was a bit too condensed ;) I fully agree with
    you on what you say about CORBA. It's just that for most people IDL
    looks a bit out of place. Especially because it resembles C. But once
    you actually wrote a few projects using CORBA, you actually begin to
    see it's elegance ;)

    I find Webservices "easier" as you can (although it's not
    recommendable) leave out the WSDL part. And some implementations
    actually generate the WSDL for you. But it's true, comparing WSDL and
    IDL I'd rather write some IDL than WSDL ;)

    But, returning back to topic, I first want to thank you all for your
    input. I appreciate all the ideas. Especially the idea of using the
    "cmd" module for sockets. That sound's very intriguing and I will very
    likely play around with that. But also PYRO seems quite useful.

    About "Linda": Am I right that it looks very similar to "JavaSpaces"?
    If yes, are there any funcdamental differences between those two?
    exhuma.twn, Feb 16, 2007
    #7
  8. > Maybe this line of mine was a bit too condensed ;) I fully agree with
    > you on what you say about CORBA. It's just that for most people IDL
    > looks a bit out of place. Especially because it resembles C. But once
    > you actually wrote a few projects using CORBA, you actually begin to
    > see it's elegance ;)


    It looks out of the place in comparison to what exactly? A WSDL-document?
    You certainly me laugh hard on that....

    > I find Webservices "easier" as you can (although it's not
    > recommendable) leave out the WSDL part. And some implementations
    > actually generate the WSDL for you.


    You can't leave WSDL out of SOAP, you can leave it out of XMLRPC. Which is
    nice and easy, but lacks any contract whatsoever. Nothing I as a Pythoneer
    care to much about, but some people bother.

    And generating the WSDL works from what exactly? ah, Java-code for example,
    using java2wsdl. Which has a striking resemblance to.... C. Funny....

    > But it's true, comparing WSDL and
    > IDL I'd rather write some IDL than WSDL ;)


    So - you say that yourself, but still you previously claimed IDL looks
    strange to the eye?

    Diez
    Diez B. Roggisch, Feb 16, 2007
    #8
  9. exhuma.twn

    Paul Boddie Guest

    On 16 Feb, 14:16, "Diez B. Roggisch" <> wrote:
    >
    > You can't leave WSDL out of SOAP


    Yes you can, since they're two different things. What you probably
    meant was that you can't leave WSDL out of "big architecture", W3C
    standards-intensive Web services. Of course, RPC-style SOAP without
    the contracts imposed by WSDL may remove some of the attractions for
    some people (who really should consider CORBA, anyway), but then
    there's always document-oriented SOAP, although if you don't want the
    baggage associated with routing and other things, plain XML messaging
    would be easier. And there's always XMPP if you want stuff like
    routing and a standard that isn't undergoing apparently continuous
    change.

    Paul
    Paul Boddie, Feb 16, 2007
    #9
  10. Paul Boddie wrote:

    > On 16 Feb, 14:16, "Diez B. Roggisch" <> wrote:
    >>
    >> You can't leave WSDL out of SOAP

    >
    > Yes you can, since they're two different things. What you probably
    > meant was that you can't leave WSDL out of "big architecture", W3C
    > standards-intensive Web services. Of course, RPC-style SOAP without
    > the contracts imposed by WSDL may remove some of the attractions for
    > some people (who really should consider CORBA, anyway), but then
    > there's always document-oriented SOAP, although if you don't want the
    > baggage associated with routing and other things, plain XML messaging
    > would be easier. And there's always XMPP if you want stuff like
    > routing and a standard that isn't undergoing apparently continuous
    > change.


    Didn't know that. Yet I presume it is pretty awful to manually decompose and
    compose the method invocations and parameter sets.

    I've got no idea what document-orientied SOAP means either. Is that just
    pushing XML-files over a protocol?

    Diez
    Diez B. Roggisch, Feb 16, 2007
    #10
  11. exhuma.twn

    Paul Boddie Guest

    On 16 Feb, 14:53, "Diez B. Roggisch" <> wrote:
    >


    [XMPP, XML messaging]

    > Didn't know that. Yet I presume it is pretty awful to manually decompose and
    > compose the method invocations and parameter sets.


    It depends on how well you like working with XML, I suppose.

    > I've got no idea what document-orientied SOAP means either. Is that just
    > pushing XML-files over a protocol?


    As I understand it, yes, although you still have to put the payload
    inside the usual SOAP boilerplate.

    Paul
    Paul Boddie, Feb 16, 2007
    #11
  12. exhuma.twn

    Steve Holden Guest

    Ben Finney wrote:
    > "Gabriel Genellina" <> writes:
    >
    >> (And I would expect that making a connection to "localhost" actually
    >> does *not* go down up to the network card hardware layer, but I
    >> don't know for real if this is the case or not).

    >
    > It damned well better. That's the entire point of the loopback
    > interface: to get all the network layer code involved, but not to talk
    > on a physical network interface.
    >
    > If a programmer decides on behalf of the user that "localhost" should
    > be treated specially, that programmer is making an error.
    >

    Inter-process TCP/IP communication between two processes on the same
    host invariably uses the loopback interface (network 127.0.0.0).
    According to standards, all addresses in that network space refer to the
    local host, though 127.0.0.1 is conventionally used.

    The transmit driver for the loopback interface receives a datagram from
    the local network layer and immediately announces its reception back to
    the local network layer.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    Blog of Note: http://holdenweb.blogspot.com
    See you at PyCon? http://us.pycon.org/TX2007
    Steve Holden, Feb 16, 2007
    #12
  13. Re: Reg Google Web Toolkit and Python

    On 2/16/07, Shadab Sayani <> wrote:
    > Hi ,
    > We have a project where I need to read files store
    > them in database in the backend.We have done this in
    > python.Now we decided to use Ajax technique for user
    > interface.For that we found that GWT is one of the
    > best toolkits.Now I got a doubt can I interface GWT
    > with python.


    http://pyjamas.pyworks.org/ is the way to go.

    --
    Roman Yakovenko
    C++ Python language binding
    http://www.language-binding.net/
    Roman Yakovenko, Feb 16, 2007
    #13
  14. > About "Linda": Am I right that it looks very similar to "JavaSpaces"?
    > If yes, are there any funcdamental differences between those two?


    Yes, they are both linda implementations, but I have no idea what so
    ever how they compare. A local java expert maybe?
    Daniel Nogradi, Feb 16, 2007
    #14
  15. exhuma.twn

    Steve Holden Guest

    Re: Reg Google Web Toolkit and Python

    Shadab Sayani wrote:
    > Hi ,
    > We have a project where I need to read files store
    > them in database in the backend.We have done this in
    > python.Now we decided to use Ajax technique for user
    > interface.For that we found that GWT is one of the
    > best toolkits.Now I got a doubt can I interface GWT
    > with python.
    > Thanks ,
    > Shadab.
    >
    > Send instant messages to your online friends http://uk.messenger.yahoo.com


    Please note that you should not create a new conversation on a newsgroup
    by editing a reply to an unrelated post - your comments are now threaded
    along wiht "Approaches of Interprocess Communication", making them
    unnecessarily hard to find and somewhat confusingly positioned.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    Blog of Note: http://holdenweb.blogspot.com
    See you at PyCon? http://us.pycon.org/TX2007
    Steve Holden, Feb 16, 2007
    #15
  16. exhuma.twn

    Steve Holden Guest

    Re: Reg Google Web Toolkit and Python

    Shadab Sayani wrote:
    > Hi ,
    > We have a project where I need to read files store
    > them in database in the backend.We have done this in
    > python.Now we decided to use Ajax technique for user
    > interface.For that we found that GWT is one of the
    > best toolkits.Now I got a doubt can I interface GWT
    > with python.
    > Thanks ,
    > Shadab.
    >
    > Send instant messages to your online friends http://uk.messenger.yahoo.com


    Please note that you should not create a new conversation on a newsgroup
    by editing a reply to an unrelated post - your comments are now threaded
    along wiht "Approaches of Interprocess Communication", making them
    unnecessarily hard to find and somewhat confusingly positioned.

    regards
    Steve
    --
    Steve Holden +44 150 684 7255 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    Blog of Note: http://holdenweb.blogspot.com
    See you at PyCon? http://us.pycon.org/TX2007
    Steve Holden, Feb 16, 2007
    #16
  17. On Fri, 16 Feb 2007 07:37:45 -0300, "Gabriel Genellina"
    <> declaimed the following in comp.lang.python:

    > > * Shared memory
    > > I don't know much about this subject.

    >
    > You forget the most basic one, stdio redirection. Easy, available on
    > almost any language, but limited to just a pair of processes.
    > You can add queues and messages.
    >


    Named pipes on those OSs that support them (applicable if both
    processes are on the same host). Even 20 year-old AmigaOS had a form of
    named pipe (more a form of virtual file <G>).

    mmap allows a form of shared memory, I believe, but doesn't really
    support some means of safe locking. Probably useful if only one process
    writes, and multiple processes are just reading (things like a satellite
    telemetry decoder filling a block of fixed status data in a continuous
    process, and other processes might be, say, a logger that runs at timed
    intervals capturing whatever current status is in the blocks along with
    a time-stamp, some control console wherein only selected status words
    are being monitored and updating a screen at some other interval...)
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
    Dennis Lee Bieber, Feb 16, 2007
    #17
  18. exhuma.twn wrote:

    > * Sockets
    > Advantage: Supported per se in nearly every programming
    > language without even the need to install additional packages
    > Disadvantage: Lot's of code to write,


    Who's Lot? :)

    No, seriously. Why would you think that it's much to write? It can,
    especially using Python, be as easy as using a file.

    > and it's kind of silly to communicate via TCP/IP if the
    > processes run on the same machine.


    Why?

    - Do you worry about CPU cycles, memory, ...?
    - Who forces you to use TCP?

    > Supposing both processes are written in Python, is there any other
    > way to achieve this?


    It depends on what those processes need to share. What you need can
    range from a simple unix socket connection to a full-fledged SQL
    database server.

    Regards,


    Björn

    --
    BOFH excuse #301:

    appears to be a Slow/Narrow SCSI-0 Interface problem
    Bjoern Schliessmann, Feb 16, 2007
    #18
  19. exhuma.twn

    Goldfish Guest

    On Feb 16, 5:11 am, "exhuma.twn" <> wrote:
    > Hi all,
    >
    > Supposing you have two separate processes running on the same box,
    > what approach would you suggest to communicate between those two
    > processes.
    >


    Spring Python makes it easy to get processes talking to each other.
    You can write your code like you are talking locally, then when its
    time to separate it either into another thread, another python
    process, or on another node, it is just a reconfiguration issue.

    http://springpython.python-hosting.com/wiki/DistributedRemoting
    Goldfish, Feb 16, 2007
    #19
  20. In article <>,
    "exhuma.twn" <> wrote:

    > Hi all,
    >
    > Supposing you have two separate processes running on the same box,
    > what approach would you suggest to communicate between those two
    > processes.


    Hi exhuma,
    That would depend on what data I was exchanging between the processes.
    For instance, if process A spawns work process B and wants to be able
    monitor B's progress, a message-based protocol might be kind of chatty.
    In this situation shared memory is probably a better fit because B can
    write it's progress to a chunk of shared memory and A can read that at
    its leisure. OTOH if the conversation is more event-driven, then a
    messaging protocol makes good sense.

    FYI there's a Python module (with sample code) for using shared memory
    on most *nix systems here:
    http://NikitaTheSpider.com/python/shm/

    HTH

    --
    Philip
    http://NikitaTheSpider.com/
    Whole-site HTML validation, link checking and more
    Nikita the Spider, Feb 16, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Swapnajit Mittra
    Replies:
    0
    Views:
    442
    Swapnajit Mittra
    Dec 21, 2004
  2. Dave Bartlett

    newbie question: interprocess communication

    Dave Bartlett, May 13, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    491
    DalePres
    May 13, 2004
  3. James  Aguilar
    Replies:
    3
    Views:
    546
    Aguilar, James
    Dec 20, 2005
  4. Michael Butscher
    Replies:
    7
    Views:
    341
    Lawrence D'Oliveiro
    Jul 1, 2006
  5. Charles Packer

    Interprocess communication: which scenario?

    Charles Packer, Dec 20, 2004, in forum: C Programming
    Replies:
    4
    Views:
    361
Loading...

Share This Page