Bidirectional named pipes on Linux

[Phil Tomson]

I've got two programs which need to communicate (which eliminates
un-named pipes). Information needs to be sent in both directions such
that one program (say it's called the master) sends a couple of bytes
to and then waits for a response from the other program (say it's called the
slave) and so on. It looks like if you open the pipe (or FIFO file) from
both sides in read/write mode that the pipe is opened non-blocking so what
ends up happening is that the side which sends first just keeps sending and
doesn't wait for a response.

....yeah, I should probably be using sockets, but this is a sort of
'legacy' system.

phil

 

[Gennady]

I've got two programs which need to communicate (which eliminates
un-named pipes). Information needs to be sent in both directions such

The mere fact that 2 independent programs need to communicate does not
eliminate un-named pipes. A third program may be written to create
un-named pipes, fork 2 processes, redirect file descriptors accordingly
in both of them and then exec your programs in those processes. It may
also be possible to exec one program in the original process.

that one program (say it's called the master) sends a couple of bytes
to and then waits for a response from the other program (say it's called the
slave) and so on. It looks like if you open the pipe (or FIFO file) from
both sides in read/write mode that the pipe is opened non-blocking so what
ends up happening is that the side which sends first just keeps sending and
doesn't wait for a response.

....yeah, I should probably be using sockets, but this is a sort of
'legacy' system.

phil

How do you specify that FIFO file to your programs? Do they know it by
name and open it explicitly or do they expect certain file descriptor to
point to the file on program startup?

You may end up with 2 FIFOs, as if you use only one FIFO a process that
wrote to the pipe and then started reading from it will compete with the
other process to which the data was intended. I would also think about
using a pty for this as it is truly bi-directional.

If you need more help, please provide details on your programs.

Gennady.

 

[Lennon Day-Reynolds]

In C, you'd use something like this:

| fcntl(fd, F_SETFL,
| fcntl(fd, F_GETFL) &
| ~O_NONBLOCK);

In Ruby, the following:

| require 'io/nonblock'
| fh = open('/tmp/myfifo')
| fh.nonblock = false

...which just hides the fcntl call and nasty constants in a nicer
interface. On my test machine, though, it looks like the FIFO opens in
blocking mode by default, at least if opened for writing.

Any of that help?

Lennon

 

[Phil Tomson]

The mere fact that 2 independent programs need to communicate does not
eliminate un-named pipes. A third program may be written to create
un-named pipes, fork 2 processes, redirect file descriptors accordingly
in both of them and then exec your programs in those processes. It may
also be possible to exec one program in the original process.

That's an idea, but in this case it probably won't work. One of the
programs is usually started much earlier than the other and stays up much
longer (the other program might be invoked and finish several times while
the first one is running).

How do you specify that FIFO file to your programs? Do they know it by
name and open it explicitly or do they expect certain file descriptor to
point to the file on program startup?

By name.

You may end up with 2 FIFOs, as if you use only one FIFO a process that
wrote to the pipe and then started reading from it will compete with the
other process to which the data was intended. I would also think about
using a pty for this as it is truly bi-directional.

Yes, that's the conclusion that I've come to: I need two FIFO files.

If you need more help, please provide details on your programs.

Sorry 'bout that. The current code is in C so i didn't want to post it
here (also it doesn't work). I'm only going to be using Ruby on one side
just to do a bit of debugging...

Phil

 

[Ara.T.Howard]

I've got two programs which need to communicate (which eliminates
un-named pipes). Information needs to be sent in both directions such
that one program (say it's called the master) sends a couple of bytes
to and then waits for a response from the other program (say it's called the
slave) and so on. It looks like if you open the pipe (or FIFO file) from
both sides in read/write mode that the pipe is opened non-blocking so what
ends up happening is that the side which sends first just keeps sending and
doesn't wait for a response.

...yeah, I should probably be using sockets, but this is a sort of
'legacy' system.

phil

what about

-------- ---- ---------
read_end fifo write_end
-------- ---- ---------
master 0 slave
slave 1 master

-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it; and a weed grows, even though we do
| not love it. --Dogen
===============================================================================

 

[Phil Tomson]

In C, you'd use something like this:

| fcntl(fd, F_SETFL,
| fcntl(fd, F_GETFL) &
| ~O_NONBLOCK);

Hmmm... are you sure about that negated O_NONBLOCK (~O_NONBLOCK)? I'm not
sure what those constants are defined as in the respective .h file, but
depending on what they are that may not work (and may not work 'badly'
.

In Ruby, the following:

| require 'io/nonblock'
| fh = open('/tmp/myfifo')
| fh.nonblock = false

..which just hides the fcntl call and nasty constants in a nicer
interface. On my test machine, though, it looks like the FIFO opens in
blocking mode by default, at least if opened for writing.

Any of that help?

I'll take a look at io/nonblock, but right now I think I actually need
two FIFOs.

(BTW: This is something I'm converting from Windows C code - apparently
they allow bidirectional pipes on Windows).

Phil

 

[Tim Sutherland]

In C, you'd use something like this:

| fcntl(fd, F_SETFL,
| fcntl(fd, F_GETFL) &
| ~O_NONBLOCK);

In Ruby, the following:

| require 'io/nonblock'
| fh = open('/tmp/myfifo')
| fh.nonblock = false

..which just hides the fcntl call and nasty constants in a nicer
interface. On my test machine, though, it looks like the FIFO opens in
blocking mode by default, at least if opened for writing.

Do you meah

fh.nonblock = true

perhaps

 

[Scott Rubin]

I've got two programs which need to communicate (which eliminates
un-named pipes). Information needs to be sent in both directions such
that one program (say it's called the master) sends a couple of bytes
to and then waits for a response from the other program (say it's called the
slave) and so on. It looks like if you open the pipe (or FIFO file) from
both sides in read/write mode that the pipe is opened non-blocking so what
ends up happening is that the side which sends first just keeps sending and
doesn't wait for a response.

...yeah, I should probably be using sockets, but this is a sort of
'legacy' system.

phil

Phil,

I'm doing this exact same thing on a linux system so that a ruby process
and a C process can communicate. It works perfectly, here is generally
how I did it. First, pipes are one way. You can have one end reading
and one end writing at the same time, but both sides may not write
simultaneously and both ends may not read simultaneously. Pipes are
strictly fifos, that's why the command to make named ones is mkfifo.
One way in, one way out. So you can either make two pipes, one for each
direction or switch who is reading and who is writing, this is difficult
if you ask me.

Next, just write to it as if it is a file. The data will stay in the
pipe until the other end opens it and then it spills out all over if you
don't catch it. It can only be read out once. So here is how I think you
should do it..

Make two fifos, one for each direction. The process that sends first
sends data down the out fifo and calls select on the in fifo. Select
will cause it to block until it is ready to read. The other process
starts out selecting on the incoming fifo and when the data finally
comes in it can send the response on the outgoing fifo.

Try to think of it like two guys with those vacuum tubes sending
messages in canisters. There is just one canister and the tubes are one
way. The guy puts the canister in the pipe that goes out and then sits
looking at the other pipe waiting for a response.

I hope this helps. I know its not bi-directional, but it really can't be.

-Scott

 

[Lennon Day-Reynolds]

Actually, since it sounded like Phil was trying to force the writer to
wait for the reader, my assumption was that he wanted to force
blocking writes.

After a bit more experimentation, though, the problem looks like it is
probably caused by buffering, rather than async writes. You could try
or'ing the O_SYNC to the flags for your open() call, or adding it
after the fact with fcntl().

Lennon

 

[Gennady]

Phil,

I'm doing this exact same thing on a linux system so that a ruby
process and a C process can communicate. It works perfectly, here is
generally how I did it. First, pipes are one way. You can have one
end reading and one end writing at the same time, but both sides may
not write simultaneously and both ends may not read simultaneously.
Pipes are strictly fifos, that's why the command to make named ones is
mkfifo. One way in, one way out. So you can

Actually, the original command for making fifos (as well as other types
of special files) is mknod (it is in /etc on some systems, in /bin or
/sbin on others):

mknod <fifo file name> p

;-)

Sincerely,
Gennady Bystritsky

 

[Phil Tomson]

Phil,

I'm doing this exact same thing on a linux system so that a ruby process
and a C process can communicate. It works perfectly, here is generally
how I did it. First, pipes are one way. You can have one end reading
and one end writing at the same time, but both sides may not write
simultaneously and both ends may not read simultaneously. Pipes are
strictly fifos, that's why the command to make named ones is mkfifo.
One way in, one way out. So you can either make two pipes, one for each
direction or switch who is reading and who is writing, this is difficult
if you ask me.

Next, just write to it as if it is a file. The data will stay in the
pipe until the other end opens it and then it spills out all over if you
don't catch it. It can only be read out once. So here is how I think you
should do it..

Make two fifos, one for each direction. The process that sends first
sends data down the out fifo and calls select on the in fifo. Select
will cause it to block until it is ready to read. The other process
starts out selecting on the incoming fifo and when the data finally
comes in it can send the response on the outgoing fifo.

Try to think of it like two guys with those vacuum tubes sending
messages in canisters. There is just one canister and the tubes are one
way. The guy puts the canister in the pipe that goes out and then sits
looking at the other pipe waiting for a response.

You wouldn't happen to have the code which shows the select call, would
you?

Phil

 

[John Carter]

It looks like if you open the pipe (or FIFO file) from
both sides in read/write mode that the pipe is opened non-blocking so what
ends up happening is that the side which sends first just keeps sending and
doesn't wait for a response.

Umm, if I understand things aright, if you open a named pipe in
read/write, mode do a write then a read, I'm afraid you just chatting to
yourself!

eg.
mkfifo foofi
ruby -e 'open("foofi", "r+") {|i| i.syswrite( "as");puts i.sysread(2)}'
as

Solution you need to open _TWO_ named pipes, one for reading, one for
writing.

Also note I used sysread and syswrite others buffering and screw up your
life.

....yeah, I should probably be using sockets, but this is a sort of
'legacy' system.

So now you have two IO's to watch, you can use a "select" to cope with
that, _or_ Threads. Did you know (Matz may have changed it but this is
the way it was when I last looked) all the threading stuff all wends it
way down into a single select lurking deep within the bowels of ruby?

ie. Ruby Thread based app === "select" event based app in a very deep
sense.

Ruby threading is just simpler.

John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand

The universe is absolutely plastered with the dashed lines exactly one
space long.