Re: named pipe and Linux

Discussion in 'Python' started by Cameron Simpson, Apr 7, 2009.

  1. On 07Apr2009 10:08, akineko <> wrote:
    | I'm trying to use named pipes to fuse a Python program and a C
    | program.
    | One side creates pipes using os.mkfifo() and both sides use the same
    | named pipes (one side reads, another side writes). The read side uses
    | select.select() to wait for incoming messages and read the message
    | when select.select() says it is ready.
    | The length of the message is unknown to the read side.

    That's a serious flaw in the message protocol.

    | I cannot use file.read() because it will block waiting for an EOF.
    | I cannot use file.readline() because how many lines have arrived is
    | unknown.
    | So, I needed to use os.read() with the exact number of characters to
    | read.

    No!

    You should use os.read() with the maximum size of a message.
    It _should_ return with the number of bytes in the message, provided the
    C program writes messages with a single OS-level write() call.

    Forget all the fstat() stuff - it's racy.

    Personally, I'd use a thread to just do continuous blocking os.read()s of
    the pipe, and putting the resulting messages on a Queue for collection
    by your main program. If you're the only consumer of a Queue it's safe
    to poll it for emptiness or not, or to use a no-wait get().

    All the above is untested, but absent a size in the protocol or other
    ways of parsing message boundaries in data stream, you can only rely on
    the C program writing messages with a single write() and collect using a
    large os.read(), which should return with what is there.

    Cheers,
    --
    Cameron Simpson <> DoD#743
    http://www.cskk.ezoshosting.com/cs/

    Language... has created the word "loneliness" to express the pain of
    being alone. And it has created the word "solitude" to express the glory
    of being alone. - Paul Johannes Tillich
     
    Cameron Simpson, Apr 7, 2009
    #1
    1. Advertising

  2. Cameron Simpson

    akineko Guest

    Hello Cameron Simpson,

    Thank you for your reply.
    I now think the way I did (using fstat()) was a very bad idea (as you
    pointed out).
    I haven't decided how to fix this yet.
    I also considered attaching the message length to the head of the
    message.
    It will work, too.
    I need to a bit more experiment.

    Thank you for providing me directions to solve this problem.

    Best regards,
    Aki Niimura

    On Apr 7, 3:26 pm, Cameron Simpson <> wrote:
    > On 07Apr2009 10:08, akineko <> wrote:
    > | I'm trying to use named pipes to fuse a Python program and a C
    > | program.
    > | One side creates pipes using os.mkfifo() and both sides use the same
    > | named pipes (one side reads, another side writes). The read side uses
    > | select.select() to wait for incoming messages and read the message
    > | when select.select() says it is ready.
    > | The length of the message is unknown to the read side.
    >
    > That's a serious flaw in the message protocol.
    >
    > | I cannot use file.read() because it will block waiting for an EOF.
    > | I cannot use file.readline() because how many lines have arrived is
    > | unknown.
    > | So, I needed to use os.read() with the exact number of characters to
    > | read.
    >
    > No!
    >
    > You should use os.read() with the maximum size of a message.
    > It _should_ return with the number of bytes in the message, provided the
    > C program writes messages with a single OS-level write() call.
    >
    > Forget all the fstat() stuff - it's racy.
    >
    > Personally, I'd use a thread to just do continuous blocking os.read()s of
    > the pipe, and putting the resulting messages on a Queue for collection
    > by your main program. If you're the only consumer of a Queue it's safe
    > to poll it for emptiness or not, or to use a no-wait get().
    >
    > All the above is untested, but absent a size in the protocol or other
    > ways of parsing message boundaries in data stream, you can only rely on
    > the C program writing messages with a single write() and collect using a
    > large os.read(), which should return with what is there.
    >
    > Cheers,
    > --
    > Cameron Simpson <> DoD#743http://www.cskk.ezoshosting.com/cs/
    >
    > Language... has created the word "loneliness" to express the pain of
    > being alone. And it has created the word "solitude" to express the glory
    > of being alone. - Paul Johannes Tillich
     
    akineko, Apr 7, 2009
    #2
    1. Advertising

  3. Cameron Simpson <> wrote:

    > On 07Apr2009 10:08, akineko <> wrote:
    >| I'm trying to use named pipes to fuse a Python program and a C
    >| program.
    >| One side creates pipes using os.mkfifo() and both sides use the same
    >| named pipes (one side reads, another side writes). The read side uses
    >| select.select() to wait for incoming messages and read the message
    >| when select.select() says it is ready.
    >| The length of the message is unknown to the read side.


    > That's a serious flaw in the message protocol.


    >| I cannot use file.read() because it will block waiting for an EOF.
    >| I cannot use file.readline() because how many lines have arrived is
    >| unknown.
    >| So, I needed to use os.read() with the exact number of characters to
    >| read.


    > No!


    > You should use os.read() with the maximum size of a message.
    > It _should_ return with the number of bytes in the message, provided the
    > C program writes messages with a single OS-level write() call.


    No!

    That's still broken. You can't know if the writer has managed to
    write one or several messages onto the pipe. And if the messages
    are large, you aren't guaranteed that the OS won't split up the
    data into multiple segments that only become available to the
    reader one or a few at a time.

    You *need* to use a protocol where it is possible to determine
    the message boundaries. You can do that by:

    - Using messages of a fixed size.

    - Terminate each message with an octet sequence that cannot occur
    within a message. For example, a linefeed without a backslash
    before it (and you would probably want a way to escape the
    backslash, in case you want to end a message with a backslash).

    - Have small header of a fixed size at the start of each message,
    that includes the length of the message in octets.


    --
    Thomas Bellman, Lysator Computer Club, Linköping University, Sweden
    "Life IS pain, highness. Anyone who tells ! bellman @ lysator.liu.se
    differently is selling something." ! Make Love -- Nicht Wahr!
     
    Thomas Bellman, Apr 8, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. richard

    named pipe problem on linux

    richard, Nov 1, 2004, in forum: C++
    Replies:
    5
    Views:
    1,035
    Sybren Stuvel
    Nov 2, 2004
  2. lee, wonsun
    Replies:
    1
    Views:
    506
    Jack Klein
    Nov 2, 2004
  3. akineko

    named pipe and Linux

    akineko, Apr 7, 2009, in forum: Python
    Replies:
    2
    Views:
    963
    bobicanprogram
    Apr 8, 2009
  4. Replies:
    3
    Views:
    219
    Ben Morrow
    Nov 28, 2007
  5. Dave Saville

    exec and named pipe questions

    Dave Saville, Sep 7, 2013, in forum: Perl Misc
    Replies:
    11
    Views:
    332
    Rainer Weikusat
    Sep 9, 2013
Loading...

Share This Page