Standard IPC for Python?

[Laszlo Nagy]

I would like to develop some module for Python for IPC. Socket
programming howto recommends that for local communication, and I
personally experienced problems with TCP (see my previous post: "Slow
network").

I was looking for semaphores and shared memory, but it is not in the
standard lib. I was also thinking about unix domain sockets, but it is
not available under windows.

I was looking for non-standard modules as well, but only found a few
references with insufficient information. For example:

http://code.activestate.com/recipes/519626/

The question is: what is the standard way to implement fast and portable
IPC with Python? Are there tools in the standard lib that can do this?


Thanks,

Laszlo

 

[Laszlo Nagy]

I use Pyro. Has always been fast enough for me. It spares you the troubles
of bloated XML-documents other RPC-mechanisms use.

Of course it is RPC, not "only" IPC - so it comes with a tradeoff. But so
far, it has been always fast enough for me.

Unfortunately, I'm developing an ORB, and using another type of ORB to
transport data would be silly. I need the lower level stuff.

 

[sturlamolden]

I would like to develop some module for Python for IPC. Socket
programming howto recommends that for local communication, and I
personally experienced problems with TCP (see my previous post: "Slow
network").

There are plenty of different IPC mechanisms available in
multiprocessing.

I was looking for semaphores and shared memory, but it is not in the
standard lib.

Take a look at:

- multiprocessing.Array and multiprocessing.Value

- mmap.mmap with 0 or -1 as the first argument (Windows and Linux,
respectively)

- pywin32 on Windows (not standard lib)

I was also thinking about unix domain sockets, but it is
not available under windows.

Unix domain sockets are called 'named pipes' under Windows. Look for
it in multiprocessing or pywin32.

The question is: what is the standard way to implement fast and portable
IPC with Python?

form multiprocessing import Queue

 

[Laszlo Nagy]

There are plenty of different IPC mechanisms available in
multiprocessing.

It is good for a special case: a tree of processes, forked from a main
process. multiprocessing.Queue cannot be used as a general message queue
between arbitrary processes.

- mmap.mmap with 0 or -1 as the first argument (Windows and Linux,
respectively)

Which lacks synchronization between processes. (How do you notify the
other process? How can the other process wait until notified?)

 

[sturlamolden]

multiprocessing.Queue cannot be used as a general message queue
between arbitrary processes.

Then e.g. use Listener and Client in multiprocessing.connection to
create a named pipe (AF_PIPE). Or use win32pipe.CreateNamedPipe from
pywin32.

- mmap.mmap with 0 or -1 as the first argument (Windows and Linux,

Which lacks synchronization between processes.

It gives you access to shared memory. You must obviously use
synchronization objects for synchronization.

(How do you notify the
other process? How can the other process wait until notified?)

multiprocessing.Event
multiprocessing.Lock
win32event.CreateEvent
win32event.SetEvent
win32event.CreateMutex

etc.