Re: select.poll.poll() never blocks

Discussion in 'Python' started by Jean-Paul Calderone, Feb 12, 2009.

  1. On Wed, 11 Feb 2009 18:44:53 -0800 (PST), birdsong <> wrote:
    >I'm pretty sure I've exhausted all searches and read all the forums
    >Google will turn up related to this issue.
    >
    >I touch an empty file in a sh shell, fire up the python shell, open
    >the file for reading(tried all buffering options), register it with a
    >poll object for select.POLLIN and call poll(), but the poll never
    >blocks and always returns for the FD, EVENT combination I ask for, but
    >the file has not anything written to it.


    Filesystem files are always reported as readable and writeable by select,
    poll, epoll, etc.

    If you want to do non-blocking filesystem I/O, your choices are to use a
    native thread (Python's threading module, or another third-party module
    which wraps the platform thread API) or to use POSIX AIO (or the mildly
    incompatible Linux variant. Of course, AIO has tons of caveats and is
    generally in a miserable state, so you probably shouldn't use it. That
    leaves you with threads.

    Jean-Paul
     
    Jean-Paul Calderone, Feb 12, 2009
    #1
    1. Advertising

  2. Jean-Paul Calderone

    birdsong Guest

    So I guess I didn't have a complete understanding of poll, I thought
    it returned file descriptors that had registered an event that the
    user asked to watch for. In my case, select.POLLIN
    Constant Meaning
    POLLIN There is data to read

    ....there ins't any data to be read. The file is readable, yes, but is
    that all that the syscall has to offer?


    On Feb 11, 6:49 pm, Jean-Paul Calderone <> wrote:
    > On Wed, 11 Feb 2009 18:44:53 -0800 (PST), birdsong <> wrote:
    > >I'm pretty sure I've exhausted all searches and read all the forums
    > >Google will turn up related to this issue.

    >
    > >I touch an empty file in a sh shell, fire up the python shell, open
    > >the file for reading(tried all buffering options), register it with a
    > >poll object for select.POLLIN and call poll(), but the poll never
    > >blocks and always returns for the FD, EVENT combination I ask for, but
    > >the file has not anything written to it.

    >
    > Filesystem files are always reported as readable and writeable by select,
    > poll, epoll, etc.
    >
    > If you want to do non-blocking filesystem I/O, your choices are to use a
    > native thread (Python's threading module, or another third-party module
    > which wraps the platform thread API) or to use POSIX AIO (or the mildly
    > incompatible Linux variant.  Of course, AIO has tons of caveats and is
    > generally in a miserable state, so you probably shouldn't use it.  That
    > leaves you with threads.
    >
    > Jean-Paul
     
    birdsong, Feb 12, 2009
    #2
    1. Advertising

  3. Jean-Paul Calderone

    Rhodri James Guest

    On Thu, 12 Feb 2009 03:01:16 -0000, birdsong <>
    wrote:

    > So I guess I didn't have a complete understanding of poll, I thought
    > it returned file descriptors that had registered an event that the
    > user asked to watch for. In my case, select.POLLIN
    > Constant Meaning
    > POLLIN There is data to read
    >
    > ...there ins't any data to be read. The file is readable, yes, but is
    > that all that the syscall has to offer?


    To quote the web page referenced in help(select):

    "It cannot be used on regular files to determine whether a file has grown
    since it was last read."

    poll() and friends are wrappers over the socket library "poll" and
    "select" functions, which primarily exist to facilitate asynchronous
    socket communications. They can take ordinary file descriptors as
    well as sockets because there's no good reason not to, but a regular
    file will always be ready to read as long as you haven't read past
    the end of file.

    You're trying to use a regular file as if it was a pipe. My first
    obvious question is do you have to do it this way? Do you have
    any control over the program writing to file? Can you cause the
    output to go to stdout and pipe it into your script's stdin
    instead? That would make your life vastly easier.

    If you can't, I'm not sure what your best strategy is. I'd
    be tempted to use "tail -f filetocheck | yourscript.py" and
    palm the job off on an already-written tool. If you want to
    do it in Python, the only thing that springs to mind is
    periodically checking the size of the file and reading more
    when that changes. You'll need to be very careful to keep
    what size you think the file is in sync with how much you've
    read!

    --
    Rhodri James *-* Wildebeeste Herder to the Masses
     
    Rhodri James, Feb 12, 2009
    #3
  4. Jean-Paul Calderone

    birdsong Guest

    On Feb 11, 7:47 pm, "Rhodri James" <>
    wrote:
    > On Thu, 12 Feb 2009 03:01:16 -0000, birdsong <>  
    > wrote:
    >
    > > So I guess I didn't have a complete understanding of poll, I thought
    > > it returned file descriptors that had registered an event that the
    > > user asked to watch for.  In my case, select.POLLIN
    > > Constant   Meaning
    > > POLLIN     There is data to read

    >
    > > ...there ins't any data to be read.  The file is readable, yes, but is
    > > that all that the syscall has to offer?

    >
    > To quote the web page referenced in help(select):
    >
    > "It cannot be used on regular files to determine whether a file has grown  
    > since it was last read."
    >
    > poll() and friends are wrappers over the socket library "poll" and
    > "select" functions, which primarily exist to facilitate asynchronous
    > socket communications.  They can take ordinary file descriptors as
    > well as sockets because there's no good reason not to, but a regular
    > file will always be ready to read as long as you haven't read past
    > the end of file.
    >
    > You're trying to use a regular file as if it was a pipe.  My first
    > obvious question is do you have to do it this way?  Do you have
    > any control over the program writing to file?  Can you cause the
    > output to go to stdout and pipe it into your script's stdin
    > instead?  That would make your life vastly easier.

    agreed, this was my original idea, but i figured implementing a tail -
    f via poll was alot faster to write and implement. i'm glad at least
    know why it wasn't working as expected.

    i've already written a tail -f similar to the one found in the link
    from a later poster, but i was hoping to use poll to cut down on
    unecessary seeks -i'm trying to avoid actions that would drive up
    iowait on the boxes this would run on.

    thanks for the clarification on the poll family of syscalls.

    >
    > If you can't, I'm not sure what your best strategy is.  I'd
    > be tempted to use "tail -f filetocheck | yourscript.py" and
    > palm the job off on an already-written tool.  If you want to
    > do it in Python, the only thing that springs to mind is
    > periodically checking the size of the file and reading more
    > when that changes.  You'll need to be very careful to keep
    > what size you think the file is in sync with how much you've
    > read!
    >
    > --
    > Rhodri James *-* Wildebeeste Herder to the Masses
     
    birdsong, Feb 12, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Arjen
    Replies:
    3
    Views:
    473
    Scott Allen
    Feb 27, 2005
  2. Soren Kuula
    Replies:
    1
    Views:
    491
    Henry S. Thompson
    Dec 1, 2005
  3. Kevin
    Replies:
    4
    Views:
    457
    Irrwahn Grausewitz
    Oct 17, 2003
  4. birdsong

    select.poll.poll() never blocks

    birdsong, Feb 12, 2009, in forum: Python
    Replies:
    2
    Views:
    472
    birdsong
    Feb 12, 2009
  5. matt
    Replies:
    1
    Views:
    297
    George Ogata
    Aug 6, 2004
Loading...

Share This Page