Python 2.6, multiprocessing module and BSD

[YouCanCallMeAl]

It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.

 

[Philip Semanchuk]

It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.

Hi Al,
I don't know anything about the multiprocessing module, but I have
some recent experience with the semaphores on FreeBSD 6 & 7 as a
result of implementing my posix_ipc module.

Since you mentioned sem_open, I assume you're talking about POSIX
semaphores, rather than System V semaphores, yes? On FreeBSD 6 & 7
sem_open exists and works, but support for said semaphores are listed
as "very experimental". I experienced a problem that looked like a bug
to me, and a rather big one at that. There's more detail here (scroll
down to the part about FreeBSD 6/7) including a link to the bug report
I filed against the FreeBSD kernel:

http://semanchuk.com/philip/posix_ipc/#platforms


HTH
Philip

 

[Jesse Noller]

It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.

The BSD issue was raised late in the cycle for 2.6. The problem is
that FBSD's support is "very experimental" as Phillip points out - and
OpenBSD doesn't even have them.

Due to the lateness of the issue and a finite amount of time I have to
work on things, I chose to disable support for this on the various
*BSDs until I can cook up a stable patch or have one provided by
someone more familiar with the inner workings of Free-BSD. OpenBSD
support is a non-starter.

Ideally, I would like to get this fixed and put on the 2.6 maint
branch ASAP, but I haven't had a chance to circle back to it.

Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
FreeBSD buildbots are so unreliable that they don't get much attention
when they go red"

Stable reliable buildbots and a few more volunteers more familiar with
BSDs might be a great and welcome addition to python-dev.

As for getting this working - I would love a patch. You are going to
want to start with python-trunk and look in setup.py. You are going to
want to adjust the flags the package uses:

elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
# FreeBSD's P1003.1b semaphore support is very experimental
# and has many known problems. (as of June 2008)
macros = dict( # FreeBSD
HAVE_SEM_OPEN=0,
HAVE_SEM_TIMEDWAIT=0,
HAVE_FD_TRANSFER=1,
)
libraries = []

You will also need to look at: Lib/multiprocessing/synchronize.py to
disable the import error - Modules/_multiprocessing/multiprocessing.h
will need to be updated for the proper ifdefs for the bsd(s) as well.
Finally, the core of the semaphore usage is in
Modules/_multiprocessing/semaphore.c

I apologize we/I could not get this in for 2.6

-jesse

 

[YouCanCallMeAl]

Hi Al,
I don't know anything about the multiprocessing module, but I have  
some recent experience with the semaphores on FreeBSD 6 & 7 as a  
result of implementing my posix_ipc module.

Since you mentioned sem_open, I assume you're talking about POSIX  
semaphores, rather than System V semaphores, yes? On FreeBSD 6 & 7  
sem_open exists and works, but support for said semaphores are listed  
as "very experimental". I experienced a problem that looked like a bug  
to me, and a rather big one at that. There's more detail here (scroll  
down to the part about FreeBSD 6/7) including a link to the bug report  
I filed against the FreeBSD kernel:

http://semanchuk.com/philip/posix_ipc/#platforms

HTH
Philip

Thanks for the background on FreeBSD semaphores. I read through the
link; it should help.

 

[YouCanCallMeAl]

It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.

The BSD issue was raised late in the cycle for 2.6. The problem is
that FBSD's support is "very experimental" as Phillip points out - and
OpenBSD doesn't even have them.

Due to the lateness of the issue and a finite amount of time I have to
work on things, I chose to disable support for this on the various
*BSDs until I can cook up a stable patch or have one provided by
someone more familiar with the inner workings of Free-BSD. OpenBSD
support is a non-starter.

Ideally, I would like to get this fixed and put on the 2.6 maint
branch ASAP, but I haven't had a chance to circle back to it.

Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
FreeBSD buildbots are so unreliable that they don't get much attention
when they go red"

Stable reliable buildbots and a few more volunteers more familiar with
BSDs might be a great and welcome addition to python-dev.

As for getting this working - I would love a patch. You are going to
want to start with python-trunk and look in setup.py. You are going to
want to adjust the flags the package uses:

        elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
            # FreeBSD's P1003.1b semaphore support is very experimental
            # and has many known problems. (as of June 2008)
            macros = dict(                  # FreeBSD
                HAVE_SEM_OPEN=0,
                HAVE_SEM_TIMEDWAIT=0,
                HAVE_FD_TRANSFER=1,
                )
            libraries = []

You will also need to look at: Lib/multiprocessing/synchronize.py to
disable the import error - Modules/_multiprocessing/multiprocessing.h
will need to be updated for the proper ifdefs for the bsd(s) as well.
Finally, the core of the semaphore usage is in
Modules/_multiprocessing/semaphore.c

I apologize we/I could not get this in for 2.6

-jesse

This is exactly the sort of response I was hoping for. Thanks for the
additional background on the problem. I'll take a look at the code and
see if I can figure out a patch. I'll also read up on the buildbot
issue ( I think I saw a link about that)...I might have a stable co-
located FreeBSD box that could be used.
-Alan

 

[Jesse Noller]

It seems that the multiprocessing module in 2.6 is broken for *BSD;
I've seen issue 3770 regarding this. I'm curious if there are more
details on this issue since the posts in 3770 were a bit unclear. For
example, one post claimed that the problem was that sem_open isn't
implemented in *BSD, but it is available on FreeBSD 7 (I checked). I'd
be willing to help get this working if someone could point me in the
right direction.

The BSD issue was raised late in the cycle for 2.6. The problem is
that FBSD's support is "very experimental" as Phillip points out - and
OpenBSD doesn't even have them.

Due to the lateness of the issue and a finite amount of time I have to
work on things, I chose to disable support for this on the various
*BSDs until I can cook up a stable patch or have one provided by
someone more familiar with the inner workings of Free-BSD. OpenBSD
support is a non-starter.

Ideally, I would like to get this fixed and put on the 2.6 maint
branch ASAP, but I haven't had a chance to circle back to it.

Also note Nick's comment in that bug: "Unfortunately, our OpenBSD and
FreeBSD buildbots are so unreliable that they don't get much attention
when they go red"

Stable reliable buildbots and a few more volunteers more familiar with
BSDs might be a great and welcome addition to python-dev.

As for getting this working - I would love a patch. You are going to
want to start with python-trunk and look in setup.py. You are going to
want to adjust the flags the package uses:

elif platform in ('freebsd5', 'freebsd6', 'freebsd7', 'freebsd8'):
# FreeBSD's P1003.1b semaphore support is very experimental
# and has many known problems. (as of June 2008)
macros = dict( # FreeBSD
HAVE_SEM_OPEN=0,
HAVE_SEM_TIMEDWAIT=0,
HAVE_FD_TRANSFER=1,
)
libraries = []

You will also need to look at: Lib/multiprocessing/synchronize.py to
disable the import error - Modules/_multiprocessing/multiprocessing.h
will need to be updated for the proper ifdefs for the bsd(s) as well.
Finally, the core of the semaphore usage is in
Modules/_multiprocessing/semaphore.c

I apologize we/I could not get this in for 2.6

-jesse

This is exactly the sort of response I was hoping for. Thanks for the
additional background on the problem. I'll take a look at the code and
see if I can figure out a patch. I'll also read up on the buildbot
issue ( I think I saw a link about that)...I might have a stable co-
located FreeBSD box that could be used.
-Alan

I know I really appreciate the additional set of eyes on this, so thanks Alan!

 

[Philip Semanchuk]

The BSD issue was raised late in the cycle for 2.6. The problem is
that FBSD's support is "very experimental" as Phillip points out - and
OpenBSD doesn't even have them.

Due to the lateness of the issue and a finite amount of time I have to
work on things, I chose to disable support for this on the various
*BSDs until I can cook up a stable patch or have one provided by
someone more familiar with the inner workings of Free-BSD. OpenBSD
support is a non-starter.

Hi Jesse,
I wasn't aware of the multiprocessing module. It looks slick! Well done.

I don't know if you clicked on the link I gave for my posix_ipc
module, but it looks like we're duplicating effort to some degree. My
module makes POSIX semaphore & shared memory primitives available to
Python programs. Obviously, what you've done is much more sophisticated.

One oversight I noticed the multiprocessing module docs is that a
semaphore's acquire() method shouldn't have a timeout on OS X as
sem_timedwait() isn't supported on that platform. (You note OS X's
lack of support for sem_getvalue() elsewhere.)

A question - how do you handle the difference in error messages on
various platforms? For instance, sem_trywait() raises error 35,
"Resource temporarily unavailable" under OS X but error 11 under
Ubuntu. Right now I'm just passing these up to the (Python) caller as
OSErrors. This makes it really hard for the Python programmer to write
cross-platform code.

The only solution I can think of (which I haven't coded yet) is to
compile & run a series of small C programs during setup.py that test
things like sem_trywait() to see what errors occur, and provide those
constants to my main .c module so that it can detect those errors
exactly and wrap them into a specific, custom error for the Python
caller.

Any thoughts on this?

Cheers
Philip

 

[Jesse Noller]

Hi Jesse,
I wasn't aware of the multiprocessing module. It looks slick! Well done.

The credit goes to R. Oudkerk, the original author of the pyprocessing
library - I'm simply a rabid user who managed to wrangle it into
Python-Core. See: http://www.python.org/dev/peps/pep-0371/

I don't know if you clicked on the link I gave for my posix_ipc module, but
it looks like we're duplicating effort to some degree. My module makes POSIX
semaphore & shared memory primitives available to Python programs.
Obviously, what you've done is much more sophisticated.

I actually saw your stuff cross the 'tubes - it looks darned nice as a
lower-level interface. What the MP package is meant to be is obviously
much more high level (and "thread like"), MP goes out of it's way to
hide the gritty internals of the semaphore management/etc - posix_ipc
is much more low level than that.

One oversight I noticed the multiprocessing module docs is that a
semaphore's acquire() method shouldn't have a timeout on OS X as
sem_timedwait() isn't supported on that platform. (You note OS X's lack of
support for sem_getvalue() elsewhere.)

Please file a ticket or update http://bugs.python.org/issue4012 so I
don't loose it, my memory is increasingly lossy. Good catch.

A question - how do you handle the difference in error messages on various
platforms? For instance, sem_trywait() raises error 35, "Resource
temporarily unavailable" under OS X but error 11 under Ubuntu. Right now I'm
just passing these up to the (Python) caller as OSErrors. This makes it
really hard for the Python programmer to write cross-platform code.

If you look at the code, we're pretty much raising OSError - it's
possible we could enhance this in later versions, but given MP is
supposed to be a cross-platform as possible and protect the user from
the seedy underbelly of semaphores/pipes/etc - when an OSError does
occur, it's generally a bug in our code, not the users.

The only solution I can think of (which I haven't coded yet) is to compile &
run a series of small C programs during setup.py that test things like
sem_trywait() to see what errors occur, and provide those constants to my
main .c module so that it can detect those errors exactly and wrap them into
a specific, custom error for the Python caller.

Any thoughts on this?

That's actually (while feeling hacky) a possibly sensible idea, the
problem is is that you'd need to maintain documentation to tell users
the exceptions for their platform.

-jesse

 

[Philip Semanchuk]

semaphore's acquire() method shouldn't have a timeout on OS X as
sem_timedwait() isn't supported on that platform. (You note OS X's
lack of
support for sem_getvalue() elsewhere.)

Please file a ticket or update http://bugs.python.org/issue4012 so I
don't loose it, my memory is increasingly lossy. Good catch.[/QUOTE]

I updated issue4012.

If you look at the code, we're pretty much raising OSError - it's
possible we could enhance this in later versions, but given MP is
supposed to be a cross-platform as possible and protect the user from
the seedy underbelly of semaphores/pipes/etc - when an OSError does
occur, it's generally a bug in our code, not the users.

Gotcha. I had a look at the code, and you're testing for errno ==
EAGAIN when sem_trywait() fails. This is correct for OS X and Ubuntu
(the platforms I mentioned above) and probably most other Unices. I
just don't have confidence that it will be true across all platforms,
esp. ones to which I don't have access like AIX and big-iron systems
that support POSIX. Maybe I am just taking defensive programming too
far.

That's actually (while feeling hacky) a possibly sensible idea, the
problem is is that you'd need to maintain documentation to tell users
the exceptions for their platform.

If I pass all errors up as OSError, yes, that will be true. But that's
actually the situation I was trying to avoid.

By generating the error at install time, I can see exactly what
platform X returns in that situation (e.g. 11). Then I can create a
#define something like this:
#define ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE 11

My main .c module can then test for errno ==
ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE and wrap that in
a custom error, like posix_ipc.SemaphoreBusyError.


Again, maybe I'm just taking defensive programming too far. I've been
bitten by other cross-platform inconsistencies at the C API level and
I'm trying to get the jump on them here.

Cheers
Philip

 

[MRAB]

I updated issue4012.



Gotcha. I had a look at the code, and you're testing for errno ==  
EAGAIN when sem_trywait() fails. This is correct for OS X and Ubuntu  
(the platforms I mentioned above) and probably most other Unices. I  
just don't have confidence that it will be true across all platforms,  
esp. ones to which I don't have access like AIX and big-iron systems  
that support POSIX. Maybe I am just taking defensive programming too  
far.





If I pass all errors up as OSError, yes, that will be true. But that's  
actually the situation I was trying to avoid.

By generating the error at install time, I can see exactly what  
platform X returns in that situation (e.g. 11). Then I can create a  
#define something like this:
#define ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE  11

My main .c module can then test for errno ==  
ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE and wrap that in  
a custom error, like posix_ipc.SemaphoreBusyError.

Again, maybe I'm just taking defensive programming too far. I've been  
bitten by other cross-platform inconsistencies at the C API level and  
I'm trying to get the jump on them here.

Re the error codes, the OS-specific code can be mapped to an OS-
agnostic(?) code. On Windows there's a Windows-specific subclass of
OSError, WindowsError, containing the Windows error code winerror as
well as errno which is inherited from OSError.