linux, named pipes (mkfifo) and a hanging application

[avishaih]

Hello all,

We (at my company) have a fairly complicated multi-threaded
application, that interfaces with user input, web (network) input and
hardware devices (the platform is a custom embedded board, running
Linux on an ARM9 CPU). To help ourselves debug and profile the
application, we've created a monitoring thread within the app that
creates two named pipes (input and output) using mkfifo(). A different
small application was created to access these pipes, thus giving us an
introspection tool in runtime. This mechanism works well when there
are no problems, meaning that while the application is running
normally, I can connect to the application using the pipes, and obtain
current status and other statistics.
My problem arises when the main app is hanging or stuck at 98-100% CPU
utilization. When at this state (which we can not find an easy way to
reproduce), the monitoring client hangs. That is, the monitoring
client is launched, but does not 'connect' to the main application. It
may be noted that other (busybox) utilities can be run from the shell,
such as 'ssh', 'ls', 'top' and friends. Also worth mentioning is that
the main app retains some of its functionality (e.g., image
streaming).
I'll be happy to supply more info as necessary,
Hope some of you gurus out there has any ideas,
Cheers,
Avishai

 

[Falcon Kirtaran]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all,

We (at my company) have a fairly complicated multi-threaded
application, that interfaces with user input, web (network) input and
hardware devices (the platform is a custom embedded board, running
Linux on an ARM9 CPU). To help ourselves debug and profile the
application, we've created a monitoring thread within the app that
creates two named pipes (input and output) using mkfifo(). A different
small application was created to access these pipes, thus giving us an
introspection tool in runtime. This mechanism works well when there
are no problems, meaning that while the application is running
normally, I can connect to the application using the pipes, and obtain
current status and other statistics.
My problem arises when the main app is hanging or stuck at 98-100% CPU
utilization. When at this state (which we can not find an easy way to
reproduce), the monitoring client hangs. That is, the monitoring
client is launched, but does not 'connect' to the main application. It
may be noted that other (busybox) utilities can be run from the shell,
such as 'ssh', 'ls', 'top' and friends. Also worth mentioning is that
the main app retains some of its functionality (e.g., image
streaming).
I'll be happy to supply more info as necessary,
Hope some of you gurus out there has any ideas,
Cheers,
Avishai

You are probably going to be yelled at because this question has very
little to do with C. Your problem is firmly within the realm of POSIX.
A book like Advanced Programming in the UNIX Environment would be very
informative about these things.

Anyway, that aside, one problem you might be having is that the
monitoring agent is not actually hanging, but blocking on reads from the
named pipe. This would happen if the application doing the image
streaming is not writing anything to the pipes. To know if this is
happening, you should check the status of the processes in top; one of
those statistics (I am pretty sure it is wa) will tell you if this is
the case.

If you want to prevent this condition, you should specify O_NONBLOCK
when opening the named pipe, and then print a message if nothing has
been read for a while. Of course, you will have to add some code to
prevent the program from going into an infinite loop around the read, if
you do this.

It would be good to inspect the code and figure out why nothing is being
written to the pipes, too. Do signal handlers constantly interrupt the
part of the program responsible for writing to the fifo?

- --
- --Falcon Darkstar Kirtaran
- --
- --OpenPGP: (7902:4457) 9282:A431

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIcBAEBAgAGBQJJa0+aAAoJEKmxP9YxEE4rxXAQALN50/0aBbqMoIU2qEyHJx3C
rPUC2KXhyK7pgo7SwF/Ay2g8DmotBb6apV1dcIqYU7oAWfTO5Rq0nS3gkVnYHlPc
YWvleXDngBM+gjyMmTcohJOU9Hv2zCyDIXuDfbJ84JGRWC/BzDd7eNCmdnCNek3I
dqHUv+bKxHB1eh/dPZ9MeXR7XsKAvs2CMO7qbOl+Ze6cKkUNQGgu1aZDwZ1fcdDO
gRN+W/geJwQ6QWBQT/l/lDtFORN/g9/cGrauljH+YeewJpaDgsva7iihLGEbUAoE
ckPThYxFu9qmKdFVyVdwhEqEIH5Lhw+IO4b3uz3aDrAGeCe+V/sCptvq8P6g/i5l
N75uv5Jb7843SuE1FeGWQTWpLdmBFvblr/0kn+e0BFsLGfRF9JVRY1xC2Wj7rr3r
S8J836dCwDSXO3Ka7V0EzdDntvCkws6zlvqyoMG+hVqOnyLCasmpCdxN/ZNpe1bz
KpxiTqD3SRKWMBpZ6Vk9AvqXdkdeJg5bCd1K+d6CWtMyWQjvsGUFUiDUzykT0Ki5
4d6OiYfXVUGChQDCYhdyYWK4KX5qeltB4ZRX8ZGPU5/dnd/HrY1WKXS7KJdMvEz5
yQUcZLN/EYzEiEHY7vi5liXeAHaYQ2bcDJM55U30t2zUyBo2JQDy4Xqf9uSOSaLH
AZ6kYnfZTuoEZbwYcPEB
=zu8A
-----END PGP SIGNATURE-----

 

[sveiki]

From your description, I can't see any easy way to figure out this
problem. Once a named pipe is created, it is usually handled like file
I/O. If your monitoring program is the one initiating requests, look
for a problem of the main program being stuck outside of the "respond
to monitor named pipe" part of the code. If you are sure that the
monitoring program opens the existing pipe properly, then it is just a
matter of finding out why the main application is no longer reading or
writing to the pipe. Sounds to me like your problem may be a normal
kind of multithreaded application issue in the main application, like
a bug causing an infinite loop (one that contains image streaming,
obviously) or possibly a data race between threads.

Really, you are going to have to use a debugger and some well-placed
logging statements to figure this problem out.

 

[Keith Thompson]

We (at my company) have a fairly complicated multi-threaded
application, that interfaces with user input, web (network) input and
hardware devices (the platform is a custom embedded board, running
Linux on an ARM9 CPU). To help ourselves debug and profile the
application, we've created a monitoring thread within the app that
creates two named pipes (input and output) using mkfifo(). A different
small application was created to access these pipes, thus giving us an
introspection tool in runtime. This mechanism works well when there
are no problems, meaning that while the application is running
normally, I can connect to the application using the pipes, and obtain
current status and other statistics.

[...]

None of what you describe is part of the standard C language or
library. Since you're running under Linux, comp.unix.programmer
or one of the Linux groups can probably be much more helpful than
we can. (There's a comp.os.linux.embedded, but I've never read it;
if you're going to post there, I suggest taking a look at it first.)

 

[Richard]

Hello all,

We (at my company) have a fairly complicated multi-threaded
application, that interfaces with user input, web (network) input and
hardware devices (the platform is a custom embedded board, running
Linux on an ARM9 CPU). To help ourselves debug and profile the
application, we've created a monitoring thread within the app that
creates two named pipes (input and output) using mkfifo(). A different
small application was created to access these pipes, thus giving us an
introspection tool in runtime. This mechanism works well when there
are no problems, meaning that while the application is running
normally, I can connect to the application using the pipes, and obtain
current status and other statistics.
My problem arises when the main app is hanging or stuck at 98-100% CPU
utilization. When at this state (which we can not find an easy way to
reproduce), the monitoring client hangs. That is, the monitoring
client is launched, but does not 'connect' to the main application. It
may be noted that other (busybox) utilities can be run from the shell,
such as 'ssh', 'ls', 'top' and friends. Also worth mentioning is that
the main app retains some of its functionality (e.g., image
streaming).
I'll be happy to supply more info as necessary,
Hope some of you gurus out there has any ideas,
Cheers,
Avishai

Did you try running it with debug info, and connecting to the process
with gdb and breaking it to see what threads are still running?

http://www.ddj.com/199200938?pgno=6

Is an introduction. But since you monitor and build this app yourself I
can only assume you have tried this approach.

 

[avishaih]

[...]

None of what you describe is part of the standard C language or
library.  Since you're running under Linux, comp.unix.programmer
or one of the Linux groups can probably be much more helpful than
we can.  (There's a comp.os.linux.embedded, but I've never read it;
if you're going to post there, I suggest taking a look at it first.)

In my humble opinion, mkfifo(), and consequently (implicit in my post,
*very* explicit in the code calls to read(), write() and others are
all very much a part of the C language. I'll still take your advice,
and post my question in a more linux-oriented group.
Cheers,
Avishai

 

[avishaih]

Did you try running it with debug info, and connecting to the process
with gdb and breaking it to see what threads are still running?

http://www.ddj.com/199200938?pgno=6

Is an introduction. But since you monitor and build this app yourself I
can only assume you have tried this approach.

You assume correctly The whole idea behind the monitoring thread
was to allow ourselves some introspection when the board is in release
mode, that is running from flash, with *very* minimal file system and
resources. I've created a board with debug version so i could use
gdbserver and connect to it, but we haven't been able to reproduce the
malfunction so far (when it does happen, it usually happens after a
few days of uptime and running).
Thanks for your interest,
Avishai

 

[Keith Thompson]

[...]

None of what you describe is part of the standard C language or
library.  Since you're running under Linux, comp.unix.programmer
or one of the Linux groups can probably be much more helpful than
we can.  (There's a comp.os.linux.embedded, but I've never read it;
if you're going to post there, I suggest taking a look at it first.)

I wrote the above. Please don't snip attribution lines.

In my humble opinion, mkfifo(), and consequently (implicit in my post,
*very* explicit in the code calls to read(), write() and others are
all very much a part of the C language. I'll still take your advice,
and post my question in a more linux-oriented group.

The C language is defined by the ISO C standard. That standard does
not define mkfifo, read, or write. Those functions are defined in a
separate standard, POSIX. There are plenty of C implementations on
non-POSIX systems.

 

[jameskuyper]

(e-mail address removed) wrote:
....

In my humble opinion, mkfifo(), and consequently (implicit in my post,
*very* explicit in the code calls to read(), write() and others are
all very much a part of the C language.

Your opinion isn't relevant; those decisions are the responsibility of
the ISO C committee, not yours. Those functions are POSIX functions,
not C functions, though they do provide a C interface to POSIX.

 

[Richard]

You assume correctly The whole idea behind the monitoring thread
was to allow ourselves some introspection when the board is in release
mode, that is running from flash, with *very* minimal file system and
resources. I've created a board with debug version so i could use
gdbserver and connect to it, but we haven't been able to reproduce the
malfunction so far (when it does happen, it usually happens after a
few days of uptime and running).
Thanks for your interest,
Avishai

Could I suggest that you find an "average" time where the pipe
connection fails and the only turn ON the monitoring app near that
time. It would help clarify one or two things from which you can move
forward.

 

[Richard]

(e-mail address removed) said:


Hmmm. This would appear to be a lie.

The source seems to be http://www.c-program.com/drichie.html - a
thirteen-question interview transcript - and the context is:

"Who are some of the people that you admire? (in the computer world
or otherwise)

Dennis Ritchie: I'm not a person who particularly had heros when
growing up."

comp.lang.c is not mentioned at all in the interview.

So - it's a lie.

One gets accustomed to trolls telling lies, of course, but in case
anyone thought Richard Noname was trustworthy, they might care to
reflect on the above.

It's not a lie. It's a joke. I will adjust it to reflect your anal
retentive objection.

 

[Antoninus Twink]

Your opinion isn't relevant; those decisions are the responsibility of
the ISO C committee, not yours.

As you've discovered, "topicality" is a sensitive question on clc.

A vocal and aggressive minority of posters want to artificially restrict
the topicality of the group to ISO C, and try to bully people into not
discussing any real-world applications of the language.

Personally, I agree with Avishai, Brian Kernigan and Dennis Richie that
read() etc. are part of the C language, as do the majority of
contributors to this group. We'll be happy to try to answer your
questions - you could also try cross-posting to comp.unix.programmer to
get the best of both groups' knowledge and experience.

 

[Kaz Kylheku]

As you've discovered, "topicality" is a sensitive question on clc.

That has nothing to do with topicality. Both the statements ``read is not part
of the C language'' and ``read is part of the C language'' are topical in
comp.lang.c. True statements about C are topical, and implicitly so are their
falsifications; debate is not possible if we exclude false statements from
being topical.

A vocal and aggressive minority of posters want to artificially restrict
the topicality of the group to ISO C, and try to bully people into not
discussing any real-world applications of the language.

In the real world, there are compilers that don't supply read and write
functions.

Personally, I agree with Avishai, Brian Kernigan and Dennis Richie that
read() etc. are part of the C language, as do the majority of

I don't believe that B.K. and D.R. think that read is part of the C language.

contributors to this group.

What constitutes the majority? Could you, say, compile a list who you
consider to be contributors, and then show (using citations) that more than 50%
of them believe that read and write are part of the C language?

Also, for bonus points, sort the list in descending order of contribution
quality, and show that this majority dominates the upper half!

 

[Antoninus Twink]

I don't believe that B.K. and D.R. think that read is part of the C language.

Interesting, then, that they discuss the read() and write() system calls
in some detail in section 8.2 of their book, which they decided to
entitle "The C programming language."

 

[jameskuyper]

I don't believe that B.K. and D.R. think that read is part of the C language.

I fully agree. However, even if they did, their opinion would be
irrelevant. They invented the language, for which we are profoundly
grateful, but responsibility for defining what the language is has
passed on to ISO; it's no longer in their hands. They are, and should
be, very influential in any discussions over such issues, and in the
unlikely event that they said anything that ridiculous, their
arguments would certainly be given very serious attention. But since
they haven't, we don't need to worry about it. The closest they have
come to doing so is to talk about the POSIX functions in the course of
discussing the C programming language.

 

[Richard]

I fully agree. However, even if they did, their opinion would be
irrelevant. They invented the language, for which we are profoundly
grateful, but responsibility for defining what the language is has
passed on to ISO; it's no longer in their hands. They are, and should

So what? This is comp.lang.c. It is NOT comp.lang.ONLY.iso.c

Like all its other language counterparts this group is dedicated to
using C in the real world to solve real issues. Not for a bunch of
pedantic empire builders to compare standard retentivity.

be, very influential in any discussions over such issues, and in the
unlikely event that they said anything that ridiculous, their
arguments would certainly be given very serious attention. But since
they haven't, we don't need to worry about it. The closest they have
come to doing so is to talk about the POSIX functions in the course of
discussing the C programming language.

As we do here. Welcome.

 

[Kaz Kylheku]

Interesting, then, that they discuss the read() and write() system calls
in some detail in section 8.2 of their book, which they decided to
entitle "The C programming language."

Even if if had been their opinion that read and write are in the C language,
that was in 1988.

In this Usenet posting, Dennis Ritchie (or else a very good forger) makes
it clear that it was the intent to describe a system interface over top of
which the standrad library could be implemented, AS AN EXAMPLE!

Here he is writing about K&R1, but the same applies to the revised K&R2:

http://groups.google.ca/group/comp.std.c/msg/b8270a17fde0c8e3
Message-ID: <[email protected]>

Here, Ritchie is following up to Dan Pop's article, in which Dan Pop opines
that open, close, read and write worked on many implementations, including
non-UNIX ones, and ought to have been standardized as part of the language,
going as far as to say that ``good pre-ANSI compilers provided evertyhing in
the K&R1, including the chapter 8 (read write stuff).''

Ritchie replies that the preface to chapter 8 makes it clear that the
chapter was intended as an example, not as part of the language to be
implemented by everyone.

The preface itself says ``[W]e will describe the basic system entry points for
I/O on the Unix system and illustrate how parts of the standard library can be
implemented with them''.

This draws a clear distinction between the standard library (i.e. that which is
in the standard language) and the entry points for I/O on the Unix system.

So in short, Dennis Ritchie does not believe that read and write are part of
the C language. He didn't believe it at the time of K&R1 (1978), K&R2 (1988),
the time of the above posting (2003), and it's unlikely that he is of a
different mind today.

 

[CBFalconer]

[...]

None of what you describe is part of the standard C language or
library. Since you're running under Linux, comp.unix.programmer
or one of the Linux groups can probably be much more helpful
than we can. (There's a comp.os.linux.embedded, but I've never
read it; if you're going to post there, I suggest taking a look
at it first.)

In my humble opinion, mkfifo(), and consequently (implicit in my
post, *very* explicit in the code calls to read(), write()
and others are all very much a part of the C language. I'll still
take your advice, and post my question in a more linux-oriented
group.

FYI c.l.c deals with the C language, as described in the various C
standards. Threads are not included in the standard, neither are
read() or write(); thus those are all off topic here. Below, the
things marked "C99" are C standards.

Some useful references about C:
<http://www.ungerhu.com/jxh/clc.welcome.txt>
<http://c-faq.com/> (C-faq)
<http://benpfaff.org/writings/clc/off-topic.html>
<http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf> (C99)
<http://cbfalconer.home.att.net/download/n869_txt.bz2> (pre-C99)
<http://www.dinkumware.com/c99.aspx> (C-library}
<http://gcc.gnu.org/onlinedocs/> (GNU docs)
<http://clc-wiki.net/wiki/C_community:comp.lang.c:Introduction>