How to get number of bytes written to nonblocking FIFO when EAGAIN is raised?

Discussion in 'Python' started by Aaron Staley, Jul 20, 2011.

  1. Aaron Staley

    Aaron Staley Guest

    Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
    operating in nonblocking mode for both read and write. Under normal
    operation all is good:

    Interpreter 1 (writer)
    >>> import os
    >>> fd = os.open('fifo', os.O_WRONLY | os.O_NONBLOCK)
    >>> f = os.fdopen(fd,'wb')
    >>> f.write('k')
    >>> f.flush()


    Interpreter 2 (reader):
    >>> import os
    >>> fd = os.open('fifo', os.O_NONBLOCK)
    >>> f = os.fdopen(fd)
    >>> f.read() #after f.flush was done in interpreter 1

    'k'

    However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)
    >>> f.write('a'*70000)

    IOError: [Errno 11] Resource temporarily unavailable

    However interpreter 2 still receives data
    >> len(f.read())

    65536

    It looks like interpreter 1 pushed data until the FIFO was full and
    then raised the IOError. Interpreter 2 constantly received some, but
    not all, of what interpreter 2 tried to send.
    Unfortunately, the IOError seems to have no attribute indicating how
    much data was successfully sent. I've looked through the docs and
    can't seem to figure out how; can anyone land some advice?

    Thanks,
    Aaron Staley
     
    Aaron Staley, Jul 20, 2011
    #1
    1. Advertising

  2. Aaron Staley

    Roy Smith Guest

    In article
    <>,
    Aaron Staley <> wrote:

    > Scenario. I have a fifo named 'fifo' on my computer (ubuntu linux)
    > operating in nonblocking mode for both read and write. Under normal
    > operation all is good:
    >
    > Interpreter 1 (writer)
    > >>> import os
    > >>> fd = os.open('fifo', os.O_WRONLY | os.O_NONBLOCK)
    > >>> f = os.fdopen(fd,'wb')

    > [...]
    > Unfortunately, the IOError seems to have no attribute indicating how
    > much data was successfully sent. I've looked through the docs and
    > can't seem to figure out how; can anyone land some advice?


    I'm thinking you want to skip the os.fdopen() call and use the file
    descriptor directly, with os.write(). I've never used this, but it
    seems like what you probably want to try.
     
    Roy Smith, Jul 20, 2011
    #2
    1. Advertising

  3. Aaron Staley

    Adam Skutt Guest

    Re: How to get number of bytes written to nonblocking FIFO whenEAGAIN is raised?

    On Jul 19, 9:19 pm, Aaron Staley <> wrote:
    > However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>>f.write('a'*70000)
    >
    > IOError: [Errno 11] Resource temporarily unavailable
    >
    > However interpreter 2 still receives data>> len(f.read())
    >
    > 65536
    >
    > It looks like interpreter 1 pushed data until the FIFO was full and
    > then raised the IOError.  Interpreter 2 constantly received some, but
    > not all, of what interpreter 2 tried to send.
    > Unfortunately, the IOError seems to have no attribute indicating how
    > much data was successfully sent.  I've looked through the docs and
    > can't seem to figure out how; can anyone land some advice?


    You need to do as Roy Smith suggested and use the actual OS I/O calls
    os.read() and os.write(). os.write() returns the number of bytes
    actually written to the underlying descriptor. Python file objects
    are akin to FILE structures in C: they perform buffering and other
    operations internally that makes them less than suitable for usage
    with asynchronous UNIX I/O. In particular, they buffer I/O internally
    before writing it to the descriptor, so there's no direct relationship
    between calling file.write() and how much data is written to the
    stream. In addition, file objects also simply raise the underlying OS
    error when it occurs. The UNIX write(2) syscall assumes that you have
    been keeping track of how many bytes you've successfully written to
    the stream and does not track it for you.

    Adam
     
    Adam Skutt, Jul 20, 2011
    #3
  4. Aaron Staley

    Aaron Staley Guest

    Re: How to get number of bytes written to nonblocking FIFO whenEAGAIN is raised?

    On Jul 19, 8:15 pm, Adam Skutt <> wrote:
    > On Jul 19, 9:19 pm, Aaron Staley <> wrote:
    >
    > > However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>> f.write('a'*70000)

    >
    > > IOError: [Errno 11] Resource temporarily unavailable

    >
    > > However interpreter 2 still receives data>> len(f.read())

    >
    > > 65536

    >
    > > It looks like interpreter 1 pushed data until the FIFO was full and
    > > then raised the IOError.  Interpreter 2 constantly received some, but
    > > not all, of what interpreter 2 tried to send.
    > > Unfortunately, the IOError seems to have no attribute indicating how
    > > much data was successfully sent.  I've looked through the docs and
    > > can't seem to figure out how; can anyone land some advice?

    >
    > You need to do as Roy Smith suggested and use the actual OS I/O calls
    > os.read() and os.write().  os.write() returns the number of bytes
    > actually written to the underlying descriptor.  Python file objects
    > are akin to FILE structures in C: they perform buffering and other
    > operations internally that makes them less than suitable for usage
    > with asynchronous UNIX I/O. In particular, they buffer I/O internally
    > before writing it to the descriptor, so there's no direct relationship
    > between calling file.write() and how much data is written to the
    > stream. In addition, file objects also simply raise the underlying OS
    > error when it occurs.  The UNIX write(2) syscall assumes that you have
    > been keeping track of how many bytes you've successfully written to
    > the stream and does not track it for you.
    >
    > Adam


    That's for the info; lower level I/O solved the issue.
    That said, is such behavior with the file objects intended? It seems
    they don't work correctly with an underlying non-blocking file
    descriptor, but the documentation doesn't state such. In fact, it
    suggests you can use non-blocking with this comment for file.read:

    Also note that when in non-blocking mode, less data than was requested
    may be returned, even if no size parameter was given.
     
    Aaron Staley, Jul 21, 2011
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page