Problem with blocking portably on sockets and Queue?

Discussion in 'Python' started by Tero Saarni, Aug 5, 2003.

  1. Tero Saarni

    Tero Saarni Guest

    Hi,

    I have several threads communicating with each other using events
    stored in Queues. Threads block on Queue.get() until somebody
    publishes an event in thread's event queue.

    I need to add support for sockets to the system. Thread needs to
    unblock when:

    - there is socket ready to be read, or
    - there is event waiting in the queue

    My first tought was to replace blocking on Queue.get() with blocking
    on poll or select and dedicating file descriptors (created with os.pipe())
    as an semaphore. Event publisher would write something to the write
    end of the pipe when it puts an event to the queue, effectively
    unblocking the receiver.

    BUT I noticed that select.poll() is not available on Windows and
    secondly Windows version of select.select() will accept only socket
    descriptors.

    What options do I have that are still portable also to Windows
    platform?

    ---
    Tero
     
    Tero Saarni, Aug 5, 2003
    #1
    1. Advertising

  2. <posted & mailed>

    Tero Saarni wrote:

    > Hi,
    >
    > I have several threads communicating with each other using events
    > stored in Queues. Threads block on Queue.get() until somebody
    > publishes an event in thread's event queue.


    So far so wonderful -- excellent architecture.


    > I need to add support for sockets to the system. Thread needs to
    > unblock when:
    >
    > - there is socket ready to be read, or
    > - there is event waiting in the queue
    >
    > My first tought was to replace blocking on Queue.get() with blocking
    > on poll or select and dedicating file descriptors (created with os.pipe())
    > as an semaphore. Event publisher would write something to the write
    > end of the pipe when it puts an event to the queue, effectively
    > unblocking the receiver.
    >
    > BUT I noticed that select.poll() is not available on Windows and
    > secondly Windows version of select.select() will accept only socket
    > descriptors.
    >
    > What options do I have that are still portable also to Windows
    > platform?


    I think you could devote a thread specifically to the task of handling
    sockets, only. That special thread would not block on any queue but
    just on select; when its select shows that a socket is ready, you can
    have the thread itself do the reading and post the appropriate special
    message including the data read to the appropriate thread, or you could
    pass the ready-for-reading socket object to the appropriate thread, or
    the like. This needs a bit more care if the set of sockets to be
    select'ed on changes with time; in this case, I think the socket-handler
    thread would have to use a timeout on its select so the set of sockets
    can be updated once in a while (either by another thread, in which
    case you need a lock to protect the "set of sockets" shared data, or
    by the socket-handling thread (SHT) itself in response to a queued
    message -- the SHT would read non-blockingly from that queue periodically
    when the select times out) -- or you could perhaps use a special
    identifiable socket for the SHT to receive from other threads requests
    to change the set of ('true') sockets to be selected on.


    Alex
     
    Alex Martelli, Aug 6, 2003
    #2
    1. Advertising

  3. Tero Saarni

    Tero Saarni Guest

    "Alex Martelli" <> wrote in message
    news:9P9Ya.42349$...
    > <posted & mailed>
    >
    > Tero Saarni wrote:
    >
    > > Hi,
    > >
    > > I have several threads communicating with each other using events
    > > stored in Queues. Threads block on Queue.get() until somebody
    > > publishes an event in thread's event queue.

    >
    > So far so wonderful -- excellent architecture.
    >
    >
    > > I need to add support for sockets to the system. Thread needs to
    > > unblock when:
    > >
    > > - there is socket ready to be read, or
    > > - there is event waiting in the queue
    > >
    > > My first tought was to replace blocking on Queue.get() with blocking
    > > on poll or select and dedicating file descriptors (created with

    os.pipe())
    > > as an semaphore. Event publisher would write something to the write
    > > end of the pipe when it puts an event to the queue, effectively
    > > unblocking the receiver.
    > >
    > > BUT I noticed that select.poll() is not available on Windows and
    > > secondly Windows version of select.select() will accept only socket
    > > descriptors.
    > >
    > > What options do I have that are still portable also to Windows
    > > platform?

    >
    > I think you could devote a thread specifically to the task of handling
    > sockets, only. That special thread would not block on any queue but
    > just on select; when its select shows that a socket is ready, you can
    > have the thread itself do the reading and post the appropriate special
    > message including the data read to the appropriate thread, or you could
    > pass the ready-for-reading socket object to the appropriate thread, or
    > the like. This needs a bit more care if the set of sockets to be
    > select'ed on changes with time; in this case, I think the socket-handler
    > thread would have to use a timeout on its select so the set of sockets
    > can be updated once in a while (either by another thread, in which
    > case you need a lock to protect the "set of sockets" shared data, or
    > by the socket-handling thread (SHT) itself in response to a queued
    > message -- the SHT would read non-blockingly from that queue periodically
    > when the select times out) -- or you could perhaps use a special
    > identifiable socket for the SHT to receive from other threads requests
    > to change the set of ('true') sockets to be selected on.
    >
    >
    > Alex
    >


    Good idea, thanks! I will try dedicating a special thread for socket
    handling, although breaking out of select() for modifying the set of
    sockets gets a little tricky as you pointed out.

    Momentarily I was even thinking that I could break out of select() by
    sending a signal when I want to modify the set of sockets, but
    as I was almost expecting, it appeared that also signal support is
    quite limited on Windows platform.

    I guess that fallback solution would be to learn Windows
    sychronization primitives (wait functions etc) and write two separate
    implementations of relevant parts of the system. With some work
    that could probably be hidden behind matching interfaces and a
    factory method choosing the implementation on the fly.

    ---
    Tero
     
    Tero Saarni, Aug 7, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendra Gunawan
    Replies:
    1
    Views:
    12,606
    Allan Herriman
    Apr 8, 2004
  2. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    689
    Russell Warren
    Jun 27, 2006
  3. nukleus
    Replies:
    14
    Views:
    840
    Chris Uppal
    Jan 22, 2007
  4. Kceiw
    Replies:
    3
    Views:
    1,001
    Jim Langston
    Mar 14, 2006
  5. Kris
    Replies:
    0
    Views:
    489
Loading...

Share This Page