What happens when your program crashes?

[jacob navia]

CB Falconer boasts:


Well, he should not read this article, since we are going to see what
happens when in the middle of the execution of your program, you see

"Segmentation fault"

And nothing. Your program doesn't exist any more, and it is up
to you to find *where* in those thousands of lines lies the fault.

First, you have to compile your program for debug, i.e. to switch on the
debug information generation. Personally I always leave that switch ON,
and only tell the linker to not include that information in the final
executable if I am going to ship the program to the customer. Otherwise,
it is always in.

Second you have to start the debugger, and run the program in the same
environment as the one where it crashed. This is important. A slight
difference in environment could mean that the crash disappears, or
that you get another crash, and not the one you were looking
for...



Let's suppose for now, that you reproduce the crash.

A "crash" is actually a signal from the CPU to the operating system
(an interrupt) that alters the flow of your program, since an interrupt
routine takes over.

This interrupt can be triggered by a wrong address, for instance, and in
the old times when I started learning C, the processor would signal that
the system bus did not accept some address. The program would stop with

"bus error".


Nowadays it is still the same. Only the appearances have changed. An
interrupt routine informs the OS that your program is trying to access
some bad address, the OS notices that the process is running under a
debugger, and then it notifies the debugger of the fact that the
program under debug crashed. The details of how this is done change
from OS to OS, but there is ONE information that MUST arrive to the
debugger (at least): the program counter.

The debugger then, looks its database of debug information in the line
numbers associative tables, and looks up where this address appears.

In the easy cases, it will find the address in question and will then be
able to display the file where the crash happened.

In many cases however, the debugger will NOT find any correspondence
between the address of the crash, and the program. In those cases, the
debugger can either
1) give up, and just display the address where the program crashed,
as gdb does some times,
2) Try to use the st*** (Oh excuse me I was just going to use that
5 letter word)... Try again
Try to use the space allocated for local variables and follow the
linked list of function activations.

One way of trying to find the current point is the brutal approach of
the lcc-win debugger. I take all locals space, and go reading all of it
trying to find out if any of the numbers in the stack correspond to any
address I have in my line number information.

If I find (as usually is the case) an integer that correspond to a
program address, I assume that this address has been left by a CALL
instruction. To verify this, I read the contents around that address
and verify that a CALL instruction is in there. If the verification
passes I am certain that a CALL was done, and that this is where
the call in the program source code is located where things went wrong.

Because, obviously, you can pass a wrong pointer to a system routine
that calls another system routine with that pointer, and the chain can
be quite deep until the pointer is actually used.

Once the debugger has the point where the last user routine was active,
it can display the source code to the user. Local variables can be
retrieved only if the debugger is able to figure out the value of
the frame pointer when this routine was called. In general this is
enormously difficult, since the system routines are highly optimized
code, without stack frames, and maybe using the frame pointer as
a normal register to hold values that have nothing to do with stack
frames, values that if followed at face value could make the debugger
crash.

---------------------------------------------------------------------

The above is a description of debugging in a workstation type
environment. In other environments, this whole stuff is much more
difficult.

Take the lcc-win debugger when used with a 16 bit DSP of Analog devices
(80-90K of RAM + 512K EPROM).

For starters, there is no possibilities of breakpoints. Eproms can be
written in 4K chunks, and setting/unsetting breakpoints like that
is out of the question.

The DSP is running a version of a FORTH virtual machine, a very
small OS/controller. The generated C then, is just instructions for
this monitor that allows for a single STOPWM instruction, that
allows to stop the program and enter the monitor.

This approach has been described by Dave Hanson et al, see reference
[1]. In the general approach of Dave Hanson, the monitor is not
FORTH based but just a small C routine.

In this case, the FORTH interpreter is used as monitor.

In this environment, there is no MMU, and writing to a wrong address
will just destroy some data elsewhere, but not provoke any
visible problem immediately. There are no crashes, at least not
of the kind we saw above.

When the program stops because of a breakpoint, the first thing the
debug monitor does, is to see if the address is in the table of 16
active breakpoints. If it is not in there, the program continues execution.

If we have reached an active breakpoint, the monitor sends a character
through the serial line to lcc-win, patiently waiting for news from the
program in the PC hooked up to the circuit board. Then, the debugger
asks for addresses of the key variables, stack contents, etc. The user
sees his/her source code, and the whole feels like visual studio

lcc-win can also send a break (holding the serial line to zero for 8 or
nine bits, I do not remember exactly). The break provokes a monitor
interrupt, and we start a debugging session like if there was
an active breakpoint.

As you can see both environments are completely different. It would be
interesting to hear from people that use debuggers in other embedded
systems to share their experiences here.

 

[Richard Heathfield]

jacob navia said:

[...] we are going to see what
happens when in the middle of the execution of your program, you see

"Segmentation fault"

And nothing. Your program doesn't exist any more, and it is up
to you to find *where* in those thousands of lines lies the fault.

Easy.
gdb ./foo core
bt

That gives you a backtrace, which will allow you to identify the point in
your program where the bug manifested itself sufficiently seriously to
cause a segfault. Then you simply work back from there.

Example:

me@here> ./foo Users "Test base 1 - users" users.sql tb1
usersstyle.css.partial > users.php
Processing table [users]
Segmentation fault (core dumped)

Hmmm, a core dump. Oh dear, how sad, never mind.

me@here> gdb ./foo core
[Version and copyright info snipped]
This GDB was configured as "i386-suse-linux"...
Core was generated by `./foo Users Test base 1 - users users.sql tb1
users'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0 0x40149ea3 in _IO_2_1_stdout_ () from /lib/libc.so.6

Hmmm. stdout? Weird. Okay...

(gdb) bt
#0 0x40149ea3 in _IO_2_1_stdout_ () from /lib/libc.so.6
#1 0x0804ff8c in pcl_DllistIterate (list=0x80660b8, extra=0xbffff414) at
pcldll.c:365
#2 0x0804cc69 in GenerateDisplay (fieldlist=0x80660b8,
prefixname=0xbffff729 "Users", fp=0x40149ea0, mdata=0xbffff4cc)
at foo.c:2026
#3 0x0804d319 in GenerateForms (fieldlist=0x80660b8, PageTitle=0xbffff72f
"Test base 1 - users",
prefixname=0xbffff729 "Users", fp=0x40149ea0, mdata=0xbffff4cc,
dbname=0xbffff75d "brc") at foo.c:2198
#4 0x0804d92b in GenSQLTableEditorCode (prefixname=0xbffff729 "Users",
PageTitle=0xbffff72f "Test base 1 - users", infilename=0xbffff753
"users.sql",
dbname=0xbffff75d "tb1", stylename=0xbffff761 "usersstyle.css.partial")
at foo.c:2331
#5 0x0804d9ce in main (argc=6, argv=0xbffff594) at foo.c:2356
#6 0x400479ed in __libc_start_main () from /lib/libc.so.6

Hmmm - pcl_DllistIterate has a good track record, so it's a fair bet that
it's been given bad data. So let's look at the caller, GenerateDisplay, at
line 2026 of my code (and yes, I know, by now I should have refactored it
into smaller files):

pcl_DllistIterate(fieldlist, &cdata);

Well, that line looks fine, so let's trace back from there a few lines.

pcl_DllistSetExecutorFunction(fieldlist, stdout);
cdata.fp = fp;
cdata.prefixname = prefixname;
cdata.mdata = *mdata;
cdata.mdata.cur_field = 0;
pcl_DllistIterate(fieldlist, &cdata);

pcl_DllistIterate iterates through the whole of a double-linked list (deep
shock there), the list being specified by its first parameter. The node
data and the second argument are both passed to a function that has been
specified in a previous call to pcl_DllistSetExecutorFunction. So there
are four likely-looking possibilities - a bad list, bad data at a
particular node, a bad 'extra' param, or a bad function.

The simplest to check is the function. (Always check the easiest option
first.) Looking a few lines up the code, we find the statement that sets
up the function, and... stdout isn't a function. Whoops. I'm so used to
passing stdout as an 'extra' that I seem to have passed it to the wrong
function by mistake. (If I'd read my diagnostics more carefully, in fact,
I'd have seen this - but I was blinded by the silly 'warning: missing
initializer' messages, of which I get dozens, and which I haven't yet
found out how to disable without also disabling lots of useful warnings.)

I change stdout to sqlfield_write_select, a real function rather than a
FILE *(!); in fact, the function I /intended/ to use for walking the list.
I re-compile. This time, I check the diagnostics more carefully. All is
well, I re-test, and the segfault is gone.

This all happened about 12 hours ago. The above is pretty much a verbatim
account of my debugging session.

Total number of lines in program (including library): 9987.
Number of lines stepped through: 0.
Number of breakpoints set: 0.
Number of watches set: 0.
Number of lines examined: 6.
Total time spent in debugger: 10-15 seconds.
Total time from segfault to bugfix: about a minute and a half.

 

[robertwessel2]

CB Falconer boasts:

 >> My point is that those units have never seen a debugger.

Well, he should not read this article, since we are going to see what
happens when in the middle of the execution of your program, you see

"Segmentation fault"

And nothing. Your program doesn't exist any more, and it is up
to you to find *where* in those thousands of lines lies the fault.

First, you have to compile your program for debug, i.e. to switch on the
debug information generation. Personally I always leave that switch ON,
and only tell the linker to not include that information in the final
executable if I am going to ship the program to the customer. Otherwise,
it is always in.

I spent a fair number of my formative years in smaller mainframe
shops. While debuggers were available, they cost money, and tended
*not* to be the sorts of things smaller mainframe shops bought.

Anyway, almost all debugging was postmortem from either inspecting the
incorrect output, or looking at a core dump (assuming the program did
the equivalent of seg faulting), and then studying the source code.
The compilers usually were set to print a map of addresses and
statements, and we'd essentially always get linkage maps as well. If
you did have a dump, you'd be able to hand disassemble the
instructions leading up to the crash, figure out what they were point
at from the compile maps, and inspect the data in question for
interesting clues. Complex issues might take considerable spelunking
through the dump.

On occasion we would, indeed, and various trace statements ("printfs")
and rerun, and some compilers had various diagnostic aids (one Cobol
compiler could collect a trace of the statements executed, which was
sometimes helpful) - although those tended to be turned off for the
production builds. But recompiling a program tending to be something
of a chore, so inserting printfs was something of a last resort. Not
to mention something of an admission of failure.

And many of these were quite substantial programs and systems.

And I still spend a good chunk of my life in places where good
debuggers are less than handy (device drivers, embedded stuff, code in
the field, etc.) - although in almost all cases something, either a
formal debugger, or something ad-hoc, can be added if necessary, in
most cases it's rather faster to not bother.

A debugger is certainly handy at times, but not essential. While work
styles vary, I see a lot of people who, IMO, have an overreliance on
debuggers (there are a couple of programmers here who seem to fire up
a debugger when their coffee gets cold). The point is to make the
best use of the available tools - which, IMO, *doesn't* include firing
up the debugger at the first sign of trouble.

 

[Doug Miller]

CB Falconer boasts:



Well, he should not read this article, since we are going to see what
happens when in the middle of the execution of your program, you see

Why do you keep splitting threads and retitling them? Is it out of ignorance,
inconsideration, or stupidity?

 

[Mark Bluemel]

Why do you keep splitting threads and retitling them? Is it out of ignorance,
inconsideration, or stupidity?

Are those mutually exclusive?

 

[Richard Heathfield]

Mark Bluemel said:

Doug Miller wrote:


Are those mutually exclusive?

No, that would be xor.

 

[jacob navia]

Are those mutually exclusive?

Of course not!

You have BOTH.

Unable to contribute *anything* to the discussion, you can only
insult people, what is still within the reach of your IQ.

You and Mr Miller are a typical example of the regulars:

1) Do not contribute anything to the discussion
2) Insult to kill discussions
3) Lower the contents of the group so that only the regulars remain.
I started that series precisely to attract people into this group,
by discussing technical matters immediately related to the C
language.

 

[jacob navia]

I spent a fair number of my formative years in smaller mainframe
shops. While debuggers were available, they cost money, and tended
*not* to be the sorts of things smaller mainframe shops bought.

Anyway, almost all debugging was postmortem from either inspecting the
incorrect output, or looking at a core dump (assuming the program did
the equivalent of seg faulting), and then studying the source code.
The compilers usually were set to print a map of addresses and
statements, and we'd essentially always get linkage maps as well. If
you did have a dump, you'd be able to hand disassemble the
instructions leading up to the crash, figure out what they were point
at from the compile maps, and inspect the data in question for
interesting clues. Complex issues might take considerable spelunking
through the dump.

Yes. But that is not the case today. Debuggers are no longer expensive.

My objective with this series is to attract young people and technically
interested people into this group. To show that C doesn't live in the
past but it has a future. Your post is interesting, but doesn't apply to
the environments of today.

On occasion we would, indeed, and various trace statements ("printfs")
and rerun, and some compilers had various diagnostic aids (one Cobol
compiler could collect a trace of the statements executed, which was
sometimes helpful) - although those tended to be turned off for the
production builds. But recompiling a program tending to be something
of a chore, so inserting printfs was something of a last resort. Not
to mention something of an admission of failure.

You were writing your "ad hoc" debugger then. This is feasible, as you
prove, but it surely is time consuming.

And many of these were quite substantial programs and systems.

And I still spend a good chunk of my life in places where good
debuggers are less than handy (device drivers, embedded stuff, code in
the field, etc.) - although in almost all cases something, either a
formal debugger, or something ad-hoc, can be added if necessary, in
most cases it's rather faster to not bother.

Yes. Writing correct code from the starts obviates a debugger.
Problem is, it is quite difficult to do.

A debugger is certainly handy at times, but not essential. While work
styles vary, I see a lot of people who, IMO, have an overreliance on
debuggers (there are a couple of programmers here who seem to fire up
a debugger when their coffee gets cold). The point is to make the
best use of the available tools - which, IMO, *doesn't* include firing
up the debugger at the first sign of trouble.

In principle this is correct. But I have gotten used to develop in the
debugger. I write a part to access some data structures, and put a
breakpoint where the calculations/usage of those data structures are
to be inserted. Then, within the debuggers, I write the new code.

 

[Bart van Ingen Schenau]

Yes. But that is not the case today. Debuggers are no longer expensive.

I beg to differ. For desktop environments debuggers might be
inexpensive, but that is not true for other environments.
I work mostly on embedded software and there debuggers are still very
expensive (upwards of EUR 10000 a piece, mostly due to the required
specialised additional hardware).
And in embedded software, I often can't use a debugger anyway because
it affects the system behaviour too much to replicate the problem.

My objective with this series is to attract young people and technically
interested people into this group. To show that C doesn't live in the
past but it has a future. Your post is interesting, but doesn't apply to
the environments of today.

Actually, it does still apply. Only not to the environments you are
using.
And incidentally, I agree that C is still live and kicking. It has
been the language that I have used most in the past 8 years and I
still use it on a daily basis.

Bart v Ingen Schenau

 

[CBFalconer]

.... snip ...

I started that series precisely to attract people into this group,
by discussing technical matters immediately related to the C
language.

Then why put it here? You should know that debuggers are OT in
this newsgroup. Just because people put up with the occasional
digression doesn't make it on-topic.

The reason is that debuggers depend on the actual implementation,
not the language. Therefore any discussion are not controlled by a
standard referance.

 

[Ian Collins]

In principle this is correct. But I have gotten used to develop in the
debugger. I write a part to access some data structures, and put a
breakpoint where the calculations/usage of those data structures are
to be inserted. Then, within the debuggers, I write the new code.

Why don't you just write unit teats?

Checking code with a debugger is all well and good, but it's a one off
and can't be automated.

 

[Richard]

Why don't you just write unit teats?

because there are not the test case writers in existence who can cover
ALL situatins and bugs DO occue wich need

a) finding
b) familiarising with
b) fixing
c) testing
d) integrating back in

Checking code with a debugger is all well and good, but it's a one off
and can't be automated.

Did you read what he said? He uses the debugger to test (as I do) as he
is writing the code. This is often a long time before any unit tests in
time constrained projects.

 

[Ian Collins]

because there are not the test case writers in existence who can cover
ALL situatins and bugs DO occue wich need

He was talking about new code, not debugging.

Did you read what he said? He uses the debugger to test (as I do) as he
is writing the code. This is often a long time before any unit tests in
time constrained projects.

Ah, well I write my tests first, especially in time constrained
projects. The less time I have to spend tracking down bugs in untested
code, the faster I work.

 

[robertwessel2]

Yes. But that is not the case today. Debuggers are no longer expensive.

My objective with this series is to attract young people and technically
interested people into this group. To show that C doesn't live in the
past but it has a future. Your post is interesting, but doesn't apply to
the environments of today.

As other have pointed out, yes, it does, many environments in which C
is used are not handy, easy, cheap, or whatever, places to run
debuggers. And as I mentioned later in my post, I spend a lot of my
time in such places.

I described the mainframe environment because your position seems to
be that such a development methodology is impossible, when, in fact,
it was very common, and widely used even by programmers of modest
talents, not all that long ago.

In principle this is correct. But I have gotten used to develop in the
debugger. I write a part to access some data structures, and put a
breakpoint where the calculations/usage of those data structures are
to be inserted. Then, within the debuggers, I write the new code.

As I mentioned, I know people who do that, and they seem to write
plenty of code. This methodology works for you, and you've clearly
written a significant chuck of code that way. Bully for you.

*I* don't like to work that way. I find the debugger to be much more
of a distraction than an asset in the ordinary course of events, and I
avoid it until I feel it's going to be a useful tool to use. I
personally find my way of doing things a superior technique, although
I'm perfectly willing to acknowledge that your way works for you.
OTOH, you've claimed that such an approach is impossible or silly, and
accused some people who claim to use that approach of being liars.
Just because you don't like that way of coding, is no reason to
dismiss it.

And just for grins, you should try Cleanroom* sometime, where not only
do you not get to use a debugger, you don't get to use the compiler
either...

*Personally I'd hate to work under those conditions

 

[jacob navia]

He was talking about new code, not debugging.

Ah, well I write my tests first, especially in time constrained
projects. The less time I have to spend tracking down bugs in untested
code, the faster I work.

You are missing most of what an integrated development environment
has to offer then.

It is easy to find out how that member of structure x was called. You
just type

x.

and then the IDE shows you the structure members list. You have just
choose the right one.

You can see the values of all key variables at the point where
you will introduce new code.

You can then, set the program counter several lines before and exercise
the new code immediately with JIT debugging (a capability lcc-win
doesn't have yet, I have to confess)

You can just right click into "go to definition" and you can see
where that variable or that function is defined.

All this makes for far less bugs than using a simple editor.

Obviously I start with the test case too. If not, I can't develop
correctly.

For instance, when I want to add a new functionality to the
compiler I start with a program that should compile, and then
work from there.

But obviously your mileage can be different.

 

[jacob navia]

OTOH, you've claimed that such an approach is impossible or silly, and
accused some people who claim to use that approach of being liars.
Just because you don't like that way of coding, is no reason to
dismiss it.

I am not dismissing it. What I just do not believe are the wild
claims being done here like Mr Heathfield saying he can debug
code over the phone without seeing it, or the others with their
stories of debugging with "just reading the code", etc.

But obviously I can debug code by reading it, and I do read
my code (and other people's ) very often. And, at the end
OBVIOUSLY the debugger can show you the source code but will
NOT tell you "the bug is in line 42"!

*YOU* have to find it,and you will find it by reading what the
debugger shows you. So I can't be against reading code.

The "super hero" stories however, I leave them to my son
(and daughter) that read a lot of comics.

Obviously reading code is a good method, as are code reviews,
and many other techniques.

But let's face it, a debugger is a real time saver. And if
you work in embedded realm, I have devloped debuggers for
very small chips (as I wrote in my previous post in this subject),
and as soon as my debugger was up and running, the programmers
immediately loved it, and it was considered *the* most important
piece of the development environment.

The theory behind it (read that article of Hanson, it is a good
read) it is very simple and can be used anywhere: a small monitor
that will allow to read selectively memory and register values
from the circuit being developed.

A serial line is all is needed!

 

[Ian Collins]

Ian Collins wrote:


You are missing most of what an integrated development environment
has to offer then.

It is easy to find out how that member of structure x was called. You
just type

x.

and then the IDE shows you the structure members list. You have just
choose the right one.

That's an IDE feature, not a debugger one. It is also one that drives
me nuts, so I turn it off. I prefer to have the header open in another
window and use two mouse clicks to copy the member over.

You can then, set the program counter several lines before and exercise
the new code immediately with JIT debugging (a capability lcc-win
doesn't have yet, I have to confess)

But that's a one off manual process. If you write a unit test to
validate the change, it becomes part of your automated test suite which
can be run each time you build.

 

[Richard Heathfield]

jacob navia said:

What I just do not believe are the wild
claims being done here like Mr Heathfield saying he can debug
code over the phone without seeing it, or the others with their
stories of debugging with "just reading the code", etc.

Not only are these "wild claims" true, but they don't even describe
anything particularly clever. Debugging unseen code is *usually*
impossible (which is why we ask people here to show us their code) - but
*sometimes* it can be done, and although it creates a great impression, it
normally only happens when the code's author is a relative newcomer to C
or to programming, and has stumbled into a common trap. Debugging code by
reading it, without a debugger - this idea that Mr Navia finds so hard to
believe, is done by clc regulars daily - *right here in clc*. I can't
imagine that any of us particularly need to fire up a debugger for most of
the problems posed here.

But obviously I can debug code by reading it,

Right - so you don't need a debugger after all. It's just a nice-to-have
for those who are good at taking advantage of its features.

But let's face it, a debugger is a real time saver.

For you, maybe. For me, it was a big time *sink*, and when I stopped using
it I saved a considerable amount of time. Your disbelief does not change
the facts.

<snip>

 

[Richard]

That's an IDE feature, not a debugger one. It is also one that drives

A debugger is frequently part of an IDE.

me nuts, so I turn it off. I prefer to have the header open in another
window and use two mouse clicks to copy the member over.

Then I would hazard that you haven't used a well configured one.

The same is true of maybe watching a certain member of a struct not
specifically mentioned in the code you are looking at.

But that's a one off manual process. If you write a unit test to
validate the change, it becomes part of your automated test suite which
can be run each time you build.

So what? You can pass all the unit tests you want and it still doesn't
stop the entire application having bugs.

 

[CBFalconer]

Why don't you just write unit teats?

Checking code with a debugger is all well and good, but it's a
one off and can't be automated.

I did something similar (debugger tests) about 50 years ago. Then
I learned better methods.