Light (general) Inter-Process Mutex/Wait/Notify Synchronization?

[Gunter Henriksen]

I am interested in the lightest mechanism to use
in a Python application for mutex/wait/notify
between processes, one of which may be a program
which does not have a shared ancestral benefactor,
or may not be a Python program at all.

I would ideally like to have something which does
not need to make a system call in an uncontended
case for the mutex.  In C on Linux I would use mmap
to get shared memory, then pthread_cond_wait with
a cond/mutex which has the PTHREAD_PROCESS_SHARED
attribute set.  For pthread_cond_wait, it would of
course be fine to make a system call.

I am familiar with the "multiprocessing" package for
the case of a closely related set of Python programs.
I am looking for something where the other side is
not a Python application using "multiprocessing".

A thin layer over pthreads would be nice, or which
is using POSIX semaphores/queues would be ok.  I
can use mmap for shared memory; that is not problem.
The synchronization is where I need help.

 

[Lawrence D'Oliveiro]

I would ideally like to have something which does
not need to make a system call in an uncontended
case for the mutex.

In Linux you could use a futex.

 

[Paul Rubin]

A thin layer over pthreads would be nice, or which
is using POSIX semaphores/queues would be ok.  I
can use mmap for shared memory; that is not problem.
The synchronization is where I need help.

Try this: http://nikitathespider.com/python/shm/

 

[Gunter Henriksen]

Try this: http://nikitathespider.com/python/shm/

I took a look at that (especially the posix_ipc at
http://semanchuk.com/philip/posix_ipc/). I am hoping
not to plug something underneath the Python VM; I
would rather use a socket, or use signals. If I were
to use a C library, I imagine I would just go with a
thin layer on top of pthread_*, but I think I will
prefer something which uses system calls and is
bundled with Python, slow as that approach may be.

 

[Paul Rubin]

I took a look at that (especially the posix_ipc at
http://semanchuk.com/philip/posix_ipc/). I am hoping not to plug
something underneath the Python VM; I would rather use a socket, or
use signals.

I'm not sure what you mean. It's just an extension module that you'd
import like any of the stdlib modules. Certainly anything involving
sockets or signals is going to be a lot slower.

 

[Philip Semanchuk]

I took a look at that (especially the posix_ipc at
http://semanchuk.com/philip/posix_ipc/).

Hej Gunter,
The posix_ipc and sysv_ipc modules both do what you're asking for. Shm
does too but sysv_ipc improves on it and has a future, unlike shm.
(I'm the author of posix_ipc and sysv_ipc and I maintain shm.)

I am hoping
not to plug something underneath the Python VM; I
would rather use a socket, or use signals. If I were
to use a C library, I imagine I would just go with a
thin layer on top of pthread_*, but I think I will
prefer something which uses system calls and is
bundled with Python, slow as that approach may be.

If you don't want to use a 3rd party module you could use the
multiprocessing module which is bundled with Python 2.6 and has been
backported to Python 2.5, I think.

You could also use the standard library's ctypes to wrap the POSIX or
SysV IPC calls. This would limit you to *nix only (or WIndows +
Cygwin) but that might be OK for you. If you go this route, I
recommend wrapping the POSIX calls rather than Sys V because the Sys V
API is a little quirky.

Good luck
Philip

 

[Gunter Henriksen]

If you don't want to use a 3rd party module you could

use the multiprocessing module

That is definitely good for when I have a tree of
processes which are all Python applications. I use
it for that. But I am looking for something where
the Python application can interact conveniently
with an arbitrary external application. Using a
socket/pipe and shared memory is convenient, but
not feasible when system call overhead matters.

 

[Gunter Henriksen]

Try this: http://nikitathespider.com/python/shm/

I'm not sure what you mean. It's just an extension module
that you'd import like any of the stdlib modules.

I cannot import it unless a corresponding extension
implementation is plugined underneath the Python VM.

Certainly anything involving sockets or signals is
going to be a lot slower.

Indeed, but I can "import signal" or "import socket"
wherever I go, without worrying about requiring (or
worse yet, having to bundle) a CPython plugin
(presuming I am even running on top of CPython).
I am hoping for something which does not require
a context switch (or worse) for every wait/notify.

 

[Aaron Brady]

That is definitely good for when I have a tree of
processes which are all Python applications.  I use
it for that.  But I am looking for something where
the Python application can interact conveniently
with an arbitrary external application.  Using a
socket/pipe and shared memory is convenient, but
not feasible when system call overhead matters.

You could write your own '.pyd' or '.so' extension that merely brokers
the OS synchronization calls. The 'ctypes' module may be enough.

You will need a way to get the owner's pid and synch. object handle to
the remote process of course.

 

[Paul Rubin]

Indeed, but I can "import signal" or "import socket"
wherever I go, without worrying about requiring (or
worse yet, having to bundle) a CPython plugin

Oh I see what you mean, yes, it's a recurring problem.

It would be nice if the mmap module included some ipc synchronization
primitives.

 

[John Nagle]

That is definitely good for when I have a tree of
processes which are all Python applications. I use
it for that. But I am looking for something where
the Python application can interact conveniently
with an arbitrary external application. Using a
socket/pipe and shared memory is convenient, but
not feasible when system call overhead matters.

Linux doesn't do interprocess communication very well.
The options are pipes (clunky), sockets (not too bad, but
excessive overhead), System V IPC (nobody uses
that) and shared memory (unsafe).

If you want to see IPC right, what you really need is
the QNX microkernel, and its messaging system. I've pumped
uncompressed video through the QNX messaging system and had
only 2% of the CPU devoted to message overhead. The basic
primitives are MsgSend, MsgReceive, and MsgReply, which do
about what you'd expect and do it very fast. Linux lacks
this.

If you're using CPython, don't worry about socket overhead.
CPython is so slow you'll never notice it.

John Nagle

 

[Gunter Henriksen]

Linux doesn't do interprocess communication very well.

The options are [...] and shared memory (unsafe).

I think the bar has to be set pretty high to say shared memory
is too unsafe an approach for active entities to communicate.

If you're using CPython, don't worry about socket overhead.
CPython is so slow you'll never notice it.

It is not so much the socket overhead as the context switch.
If processes could send data to each other through sockets
without making system calls, that would be great.

 

[JanC]

Linux doesn't do interprocess communication very well.
The options are pipes (clunky), sockets (not too bad, but
excessive overhead), System V IPC (nobody uses
that) and shared memory (unsafe).

+ dbus

 

[Lawrence D'Oliveiro]

Linux doesn't do interprocess communication very well.
... and shared memory (unsafe).

What about with a futex?