multiprocessing on freebsd

Discussion in 'Python' started by Tim Arnold, Mar 17, 2010.

  1. Tim Arnold

    Tim Arnold Guest

    Hi,
    I'm checking to see if multiprocessing works on freebsd for any
    version of python. My server is about to get upgraded from 6.3 to 8.0
    and I'd sure like to be able to use multiprocessing.

    I think the minimal test would be:
    ---------------------
    import multiprocessing
    q = multiprocessing.Queue()
    ---------------------

    with 6.3, I get

    File "/usr/local/lib/python2.6/multiprocessing/__init__.py", line
    212, in Queue
    from multiprocessing.queues import Queue
    File "/usr/local/lib/python2.6/multiprocessing/queues.py", line 22,
    in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore,
    Semaphore, Condition
    File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line
    33, in <module>
    " function, see issue 3770.")
    ImportError: This platform lacks a functioning sem_open
    implementation, therefore, the required synchronization primitives
    needed will not function, see issue 3770.


    thanks for any info,
    --Tim Arnold
    Tim Arnold, Mar 17, 2010
    #1
    1. Advertising

  2. On Mar 17, 2010, at 9:30 AM, Tim Arnold wrote:

    > Hi,
    > I'm checking to see if multiprocessing works on freebsd for any
    > version of python. My server is about to get upgraded from 6.3 to 8.0
    > and I'd sure like to be able to use multiprocessing.
    >
    > I think the minimal test would be:
    > ---------------------
    > import multiprocessing
    > q = multiprocessing.Queue()
    > ---------------------
    >
    > with 6.3, I get
    >
    > File "/usr/local/lib/python2.6/multiprocessing/__init__.py", line
    > 212, in Queue
    > from multiprocessing.queues import Queue
    > File "/usr/local/lib/python2.6/multiprocessing/queues.py", line 22,
    > in <module>
    > from multiprocessing.synchronize import Lock, BoundedSemaphore,
    > Semaphore, Condition
    > File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line
    > 33, in <module>
    > " function, see issue 3770.")
    > ImportError: This platform lacks a functioning sem_open
    > implementation, therefore, the required synchronization primitives
    > needed will not function, see issue 3770.


    Hi Tim,
    Under FreeBSD 8/Python 2.6.2 I get the same result, unfortunately.
    That's a pity because sem_open works under FreeBSD >= 7.2 as we
    discussed before.


    Issue 3770 is closed with the note, "we've removed hard-coded platform
    variables for a better autoconf approach." I'm using the Python built
    from FreeBSD's ports, and the note makes me think that it's possible
    that if I built my own Python from the Python.org tarball rather than
    ports the problem would go away due to autoconf magic. I don't have
    the time to offer to do this for you, unfortunately. But why not
    install FreeBSD 8 under VirtualBox or somesuch and give it a go
    yourself?

    A couple of quirks I noted related to FreeBSD & POSIX IPC that you
    might find useful --
    - The sem and mqueuefs kernel modules must be loaded, otherwise you'll
    get a message like this when you try to create a semaphore or message
    queue:
    Bad system call: 12 (core dumped)

    Under 8.0 they're loaded by default, I think.

    - C apps that want to use message queues must link to the realtime
    libs (pass -lrt to the linker). This tripped me up for a while.
    Linking to the realtime libs is required for all POSIX IPC calls under
    Linux; FreeBSD does not require it for semaphores or shared mem, only
    message queues.


    Hope this helps
    Philip
    Philip Semanchuk, Mar 17, 2010
    #2
    1. Advertising

  3. Tim Arnold

    Tim Arnold Guest

    On Mar 17, 11:26 am, Philip Semanchuk <> wrote:
    > On Mar 17, 2010, at 9:30 AM, Tim Arnold wrote:
    >
    >
    >
    >
    >
    > > Hi,
    > > I'm checking to see if multiprocessing works on freebsd for any
    > > version of python. My server is about to get upgraded from 6.3 to 8.0
    > > and I'd sure like to be able to use multiprocessing.

    >
    > > I think the minimal test would be:
    > > ---------------------
    > > import multiprocessing
    > > q = multiprocessing.Queue()
    > > ---------------------

    >
    > > with 6.3, I get

    >
    > > File "/usr/local/lib/python2.6/multiprocessing/__init__.py", line
    > > 212, in Queue
    > >    from multiprocessing.queues import Queue
    > >  File "/usr/local/lib/python2.6/multiprocessing/queues.py", line 22,
    > > in <module>
    > >    from multiprocessing.synchronize import Lock, BoundedSemaphore,
    > > Semaphore, Condition
    > >  File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line
    > > 33, in <module>
    > >    " function, see issue 3770.")
    > > ImportError: This platform lacks a functioning sem_open
    > > implementation, therefore, the required synchronization primitives
    > > needed will not function, see issue 3770.

    >
    > Hi Tim,
    > Under FreeBSD 8/Python 2.6.2 I get the same result, unfortunately.  
    > That's a pity because sem_open works under FreeBSD >= 7.2 as we  
    > discussed before.
    >
    > Issue 3770 is closed with the note, "we've removed hard-coded platform  
    > variables for a better autoconf approach." I'm using the Python built  
    > from FreeBSD's ports, and the note makes me think that it's possible  
    > that if I built my own Python from the Python.org tarball rather than  
    > ports the problem would go away due to autoconf magic. I don't have  
    > the time to offer to do this for you, unfortunately. But why not  
    > install FreeBSD 8 under VirtualBox or somesuch and give it a go  
    > yourself?
    >
    > A couple of quirks I noted related to FreeBSD & POSIX IPC that you  
    > might find useful --
    > - The sem and mqueuefs kernel modules must be loaded, otherwise you'll  
    > get a message like this when you try to create a semaphore or message  
    > queue:
    > Bad system call: 12 (core dumped)
    >
    > Under 8.0 they're loaded by default, I think.
    >
    > - C apps that want to use message queues must link to the realtime  
    > libs (pass -lrt to the linker). This tripped me up for a while.  
    > Linking to the realtime libs is required for all POSIX IPC calls under  
    > Linux; FreeBSD does not require it for semaphores or shared mem, only  
    > message queues.
    >
    > Hope this helps
    > Philip


    Hi Philip,
    Thanks for that information (esp the linker info). Once the machine is
    upgraded, I'll try building python from the tarball. I'll post back
    here with the results.

    here's hoping!
    thanks,
    --Tim
    Tim Arnold, Mar 17, 2010
    #3
  4. On 03/17/10 13:30, Tim Arnold wrote:
    > Hi,
    > I'm checking to see if multiprocessing works on freebsd for any
    > version of python. My server is about to get upgraded from 6.3 to 8.0
    > and I'd sure like to be able to use multiprocessing.
    >
    > I think the minimal test would be:
    > ---------------------
    > import multiprocessing
    > q = multiprocessing.Queue()
    > ---------------------
    >
    > with 6.3, I get
    >
    > File "/usr/local/lib/python2.6/multiprocessing/__init__.py", line
    > 212, in Queue
    > from multiprocessing.queues import Queue
    > File "/usr/local/lib/python2.6/multiprocessing/queues.py", line 22,
    > in<module>
    > from multiprocessing.synchronize import Lock, BoundedSemaphore,
    > Semaphore, Condition
    > File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line
    > 33, in<module>
    > " function, see issue 3770.")
    > ImportError: This platform lacks a functioning sem_open
    > implementation, therefore, the required synchronization primitives
    > needed will not function, see issue 3770.
    >


    Build mine from ports, with the following options (notice SEM & PTH):
    [martin@aspire8930 /usr/home/martin]$ cat /var/db/ports/python26/options
    # This file is auto-generated by 'make config'.
    # No user-servicable parts inside!
    # Options for python26-2.6.4
    _OPTIONS_READ=python26-2.6.4
    WITH_THREADS=true
    WITHOUT_HUGE_STACK_SIZE=true
    WITH_SEM=true
    WITH_PTH=true
    WITH_UCS4=true
    WITH_PYMALLOC=true
    WITH_IPV6=true
    WITHOUT_FPECTL=true

    [martin@aspire8930 /usr/home/martin]$ uname -a
    FreeBSD aspire8930 8.0-STABLE FreeBSD 8.0-STABLE #3: Wed Feb 3 17:01:18
    GMT 2010 martin@aspire8930:/usr/obj/usr/src/sys/ASPIRE8930 amd64
    [martin@aspire8930 /usr/home/martin]$ python
    Python 2.6.4 (r264:75706, Mar 17 2010, 18:44:24)
    [GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import multiprocessing as mp
    >>> queue = mp.Queue()
    >>>


    hth
    --
    mph
    Martin P. Hellwig, Mar 17, 2010
    #4
  5. Tim Arnold

    Tim Arnold Guest

    "Martin P. Hellwig" <> wrote in message
    news:hnrabj$c4v$-september.org...
    > On 03/17/10 13:30, Tim Arnold wrote:
    >> Hi,
    >> I'm checking to see if multiprocessing works on freebsd for any
    >> version of python. My server is about to get upgraded from 6.3 to 8.0
    >> and I'd sure like to be able to use multiprocessing.
    >>
    >> I think the minimal test would be:
    >> ---------------------
    >> import multiprocessing
    >> q = multiprocessing.Queue()
    >> ---------------------
    >>
    >> with 6.3, I get
    >>
    >> File "/usr/local/lib/python2.6/multiprocessing/__init__.py", line
    >> 212, in Queue
    >> from multiprocessing.queues import Queue
    >> File "/usr/local/lib/python2.6/multiprocessing/queues.py", line 22,
    >> in<module>
    >> from multiprocessing.synchronize import Lock, BoundedSemaphore,
    >> Semaphore, Condition
    >> File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line
    >> 33, in<module>
    >> " function, see issue 3770.")
    >> ImportError: This platform lacks a functioning sem_open
    >> implementation, therefore, the required synchronization primitives
    >> needed will not function, see issue 3770.
    >>

    >
    > Build mine from ports, with the following options (notice SEM & PTH):
    > [martin@aspire8930 /usr/home/martin]$ cat /var/db/ports/python26/options
    > # This file is auto-generated by 'make config'.
    > # No user-servicable parts inside!
    > # Options for python26-2.6.4
    > _OPTIONS_READ=python26-2.6.4
    > WITH_THREADS=true
    > WITHOUT_HUGE_STACK_SIZE=true
    > WITH_SEM=true
    > WITH_PTH=true
    > WITH_UCS4=true
    > WITH_PYMALLOC=true
    > WITH_IPV6=true
    > WITHOUT_FPECTL=true
    >
    > [martin@aspire8930 /usr/home/martin]$ uname -a
    > FreeBSD aspire8930 8.0-STABLE FreeBSD 8.0-STABLE #3: Wed Feb 3 17:01:18
    > GMT 2010 martin@aspire8930:/usr/obj/usr/src/sys/ASPIRE8930 amd64
    > [martin@aspire8930 /usr/home/martin]$ python
    > Python 2.6.4 (r264:75706, Mar 17 2010, 18:44:24)
    > [GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
    > Type "help", "copyright", "credits" or "license" for more information.
    > >>> import multiprocessing as mp
    > >>> queue = mp.Queue()
    > >>>

    >
    > hth
    > --
    > mph


    Hi Martin, thanks very much for posting that. All I can say is YAY! I'm
    really looking forward to my machine's upgrade now!

    thanks,
    --Tim
    Tim Arnold, Mar 18, 2010
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Victor Bazarov
    Replies:
    0
    Views:
    494
    Victor Bazarov
    Jul 4, 2003
  2. Ravi I Singh
    Replies:
    1
    Views:
    472
    Rolf Magnus
    Oct 13, 2003
  3. Charles

    Program for FreeBSD

    Charles, Nov 10, 2003, in forum: C++
    Replies:
    3
    Views:
    374
    Thomas Matthews
    Nov 10, 2003
  4. Charles Prince
    Replies:
    0
    Views:
    331
    Charles Prince
    Dec 6, 2004
  5. Tim Arnold

    freebsd and multiprocessing

    Tim Arnold, Mar 2, 2010, in forum: Python
    Replies:
    5
    Views:
    1,299
    Pop User
    Mar 2, 2010
Loading...

Share This Page