Debugging in the large, modern practice?

[Hugh Sasse]

I think the following may be a badly formed question, but if you'd
bear with me....

I have a large application (which is actually a Rails app) which is
behaving oddly (I can change items in a DB twice, but 4 times
fails), and using all the conventional approaches I have learned for
debugging (printing things out, logging to files, ...) it is taking
me an age to track the problem down. I have no good reason to assert
that the database or Rails is at fault, it is more likely to be my
code, but the interactions with the other code make debugging more
difficult.

So, my question is this: Given that since I started working in
computing there have been major strides in software development,
such as Object Oriented programming becoming mainstream, development
of concepts like refactoring, development of practices such as the
Agile methodologies, not to mention developments in networking and
databases, what are the parallel developments in debugging large
systems? By large, I mean sufficiently large to cause problems in
the mental modelling of the dynamic nature of the process, and
involving considerable quantities of other people's code.

The experience I have gained seems to be insufficient to meet the
kinds of demands that cannot be unique to my situation, so there
must be better approaches out there already if others are meeting
such demands.

Given the prevalence of metaprogramming in Ruby, I'll phrase this
another way, as a meta-question: what are good questions to ask to
progress along the road of improving one's ability to debug large
systems?

Thank you,
Hugh

 

[Andrew Stewart]

So, my question is this: Given that since I started working in
computing there have been major strides in software development,
such as Object Oriented programming becoming mainstream, development
of concepts like refactoring, development of practices such as the
Agile methodologies, not to mention developments in networking and
databases, what are the parallel developments in debugging large
systems? By large, I mean sufficiently large to cause problems in
the mental modelling of the dynamic nature of the process, and
involving considerable quantities of other people's code.

Debugging large systems is indeed hard -- lots to remember at once
with too many moving parts. It's much easier to debug small things.

Alongside all the other strides you mention, testing has improved no
end as I am sure you are aware. Tools like autotest [1] make it easy
to run tests against your system all the time, so you notice sooner
rather than later when it doesn't behave the way you (via your tests)
expect it to.

It's daunting when confronted with a large pile of someone else's
code, especially if that code doesn't have tests, but you have to
start somewhere. You can write tests against the third-party code
which over time become your own personal (executable) documentation
of its API.

With your specific problem, perhaps you could write a test for the
operation you are trying to do. Start with one that passes. Now add
more tests until one fails -- should be easy as it sounds like you
can reliably make the system fail. Now you can iteratively try to
write intermediate tests between the one that passes and the one that
fails, until you isolate the problem to a very small change
somewhere. Think of it as a binary search via tests of the problem
space. Hopefully you will be able to converge on the problematic
needle in the haystack of code.

And then when you have isolated and fixed the problem, you have a
nice set of tests which ensure it won't reappear later on -- a
benefit which manually inspecting log files doesn't confer.

You probably know all this already, so apologies ;-)

Good luck,
Andy Stewart


[1] http://www.zenspider.com/ZSS/Products/ZenTest/

 

[Michael Glaesemann]

what are the parallel developments in debugging large
systems? By large, I mean sufficiently large to cause problems in
the mental modelling of the dynamic nature of the process, and
involving considerable quantities of other people's code.

Though I can't speak to your larger question as to history of
debugging and its progress, I'm finding that writing (unit,
functional) tests is really helping me understand my own applications
better as well as understand the frameworks and other code they rely
on. Tests can fail unexpectedly, and digging a bit to find the reason
for the exhibited behavior can be very edifying. And as the scope of
the test decreases, so does the number of parts that are contributing
to the behavior. I've just had a good couple of days doing nothing
but writing unit tests and I'm really happy with how I feel more
comfortable with the code and its behavior.

Michael Glaesemann
grzm seespotcode net

 

[Eivind Eklund]

So, my question is this: Given that since I started working in
computing there have been major strides in software development,
such as Object Oriented programming becoming mainstream, development
of concepts like refactoring, development of practices such as the
Agile methodologies, not to mention developments in networking and
databases, what are the parallel developments in debugging large
systems?

Invariant checks ("Design by Contract"), mostly. Adding invariant
checks to your system will often unearth weird cases - making them
easily reproducable and making them fail early.

Testing (which has already been mentioned) can handle some of the same
niche, by removing coupling and allowing you to externally check for
invariants being kept. However, it is usually much easier to add
invariant checks to a system than to add tests.

Eivind.

 

[James Edward Gray II]

With your specific problem, perhaps you could write a test for the
operation you are trying to do. Start with one that passes. Now
add more tests until one fails -- should be easy as it sounds like
you can reliably make the system fail. Now you can iteratively try
to write intermediate tests between the one that passes and the one
that fails, until you isolate the problem to a very small change
somewhere.

Plus, when you find the one that fails you will no every subsystem
involved up to there. You can begin to go write tests for those too
now, ensuring they function as expected too. Hopefully you would
slowly zero in on the problem.

James Edward Gray II

 

[Richard Conroy]

I have a large application (which is actually a Rails app) which is
behaving oddly (I can change items in a DB twice, but 4 times
fails), and using all the conventional approaches I have learned for
debugging (printing things out, logging to files, ...)

I am not being pedantic here, but have you not tried a debugger?
Standard out is ok for only certain tasks, particularly if there is
an extended time aspect to the problem.

... what are the parallel developments in debugging large
systems? By large, I mean sufficiently large to cause problems in
the mental modelling of the dynamic nature of the process, and
involving considerable quantities of other people's code.

-Remote debugging (ie. servet, web service, .NET debugging in your
IDE as if it was a locally running program)
-Automated Unit Testing (JUnit, NUnit, RUnit)
-Unit Test coverage measuring (rcov, JCoverage, Clover)
-Automated UI testing (of web UIs, excellent examples being WATIR,
Selenium etc.)

If you are getting something as fundamental as somethign screwing up
your DB saves, I would probably start looking at whether you are
properly covered by your unit tests. If you find gaping holes in your
test coverage I would write more tests to expose the problem.

Given the prevalence of metaprogramming in Ruby, I'll phrase this
another way, as a meta-question: what are good questions to ask to
progress along the road of improving one's ability to debug large
systems?

Simple answer: break up large systems into smaller systems
that can be tested independently (and possibly even replaceable). Its
not so much a component concept as just the age old practice of
modular systems.

I am not sure how this might apply to large Ruby apps, my gut feeling
is there are not many huge ruby apps out there. The biggest I have
heard is around 30000 LOC, which is an awful lot for Ruby. Equivalent
functionality in a more conventional language (Java, C++, NET etc.)
would be bigger.

Rails apps are a bit different though, as Rails is an opinionated framework,
and logically dictates where your code should be. You also don't end up
writing much Ruby code. If you are, you're probably not leaning on the
stack enough. My gut feeling is that code bloat in Rails will very quickly
reveal itself that you are doing something wrong (possibly when test code
starts to get difficult to write). To get 30000 LOC in a Rails app, when it
is written well, your app would be huge, massively featured and have a
pretty huge UI (lots of RHTML), it might be possible to achieve that if you
are localising views a lot (while globalise nicely lets you avoid this as
much as possible, things like date controls, right->left reading order,
might make it easier to localise at the template level.

 

[M. Edward (Ed) Borasky]

I think the following may be a badly formed question, but if you'd
bear with me....

I have a large application (which is actually a Rails app) which is
behaving oddly (I can change items in a DB twice, but 4 times
fails), and using all the conventional approaches I have learned for
debugging (printing things out, logging to files, ...) it is taking
me an age to track the problem down. I have no good reason to assert
that the database or Rails is at fault, it is more likely to be my
code, but the interactions with the other code make debugging more
difficult.

A couple of questions:

1. How large is "large"? Is there some kind of "code size metric" that
the Ruby community uses, and a tool or tools to measure it?

2. You say "your code". How much of this application have you personally
written, how much is "Rails and Ruby and the rest of the
infrastructure", and how much is "the rest of it"?

So, my question is this: Given that since I started working in
computing there have been major strides in software development,
such as Object Oriented programming becoming mainstream, development
of concepts like refactoring, development of practices such as the
Agile methodologies, not to mention developments in networking and
databases, what are the parallel developments in debugging large
systems? By large, I mean sufficiently large to cause problems in
the mental modelling of the dynamic nature of the process, and
involving considerable quantities of other people's code.

The "traditional CASE tools" -- IDEs, software configuration and project
management tool sets, the waterfall model, the CMM levels, and of course
prayer, threats, outsourcing and pizza.

The experience I have gained seems to be insufficient to meet the
kinds of demands that cannot be unique to my situation, so there
must be better approaches out there already if others are meeting
such demands.

Again, without knowing both the scope of your specific project nor the
size of the team that built/is building it, it's difficult to answer.
There are bazillions of failed silver bullets to choose from. My
personal opinion is that you're being too hard on yourself and that
anybody who claims to have a tool or a process or a programming language
that is *significantly* better than today's common practices is either
deceived, deceiving or both.

Given the prevalence of metaprogramming in Ruby, I'll phrase this
another way, as a meta-question: what are good questions to ask to
progress along the road of improving one's ability to debug large
systems?

I think first of all, you have to *want* to debug large chunks of other
peoples' code. It's an acquired taste. I acquired it at one point in my
career but found it unsatisfying. If you *don't* want to debug large
chunks of other peoples' code, there are ways you can structure your
team and processes to minimize how much of it you have to do.

And I would caution you that, although testing and test-driven
development are certainly important and worthwhile, testing can only
show the *presence* of defects, not the absence of defects.

At the point in my career when I was at the peak of my ability to debug
other peoples' code, I came up with a simple rule. You'll probably need
to adjust the time scales to suit your situation, but in my case, the
rule was: If I don't find the problem in one day, it's not my mistake,
but a mistake in someone else's work. And if it takes more than a week,
it's not a software problem, it's a hardware problem.

Good luck ... may the source be with you.

 

[Kent Sibilev]

I have a large application (which is actually a Rails app) which is
behaving oddly (I can change items in a DB twice, but 4 times
fails), and using all the conventional approaches I have learned for
debugging (printing things out, logging to files, ...) it is taking
me an age to track the problem down. I have no good reason to assert
that the database or Rails is at fault, it is more likely to be my
code, but the interactions with the other code make debugging more
difficult.

My first bet would be on improving the test case suite. But in the
process I think that a good debugger is a very valuable tool too. You
can try ruby-debug, which I find very helpful sometimes:

http://datanoise.com/articles/2006/09/14/debugging-rails-application

 

[Hugh Sasse]

I've delayed replying for a bit because lots of good stuff is
coming in, and I didn't want to stem the flow or divert it, ...

I am not being pedantic here, but have you not tried a debugger?

I'm not yet clear where to set the breakpoints, and so on, because
I don't understand enough about the nature of the failure. But
I probably need to re-examine the Ruby debugger, because my knowledge
of it's capabilities is somewhat limited.... So, up to now, no,
I have avoided this approach.

Standard out is ok for only certain tasks, particularly if there is
an extended time aspect to the problem.

Hence the logging...

-Remote debugging (ie. servet, web service, .NET debugging in your
IDE as if it was a locally running program)

I'm not on .NET, I'm on Solaris and also Cygwin. Now, some
people will want to spit molten sulphur at me for using Cygwin.
Given that I'm more fluent in Unix than Windows, I'd be glad of
an alternative if there is one, but I don't think that's the
problem here anyway. Oh, and the usual vanishingly
small budget constraints for us seem to apply. "There might be
money next year"

-Automated Unit Testing (JUnit, NUnit, RUnit)
-Unit Test coverage measuring (rcov, JCoverage, Clover)
-Automated UI testing (of web UIs, excellent examples being WATIR,
Selenium etc.)

Using Watir already. Other testing could do with improvment.

If you are getting something as fundamental as somethign screwing up
your DB saves, I would probably start looking at whether you are

Well, the logs show the save! commands are getting called OK, with
the right values. I've said I've no reason to suspect the database.
Leaving reason aside, my emotions tell me that maybe I should
suspect, having read rather uncomplimentary stuff about MySQL,
advocating PostgreSQL, but I don't know enough about the DB issues
to be sure. This is my first serious DB project. So for now I
suspect my own code. So I'll need to refactor to improve testing.

properly covered by your unit tests. If you find gaping holes in your
test coverage I would write more tests to expose the problem.


Simple answer: break up large systems into smaller systems
that can be tested independently (and possibly even replaceable). Its
not so much a component concept as just the age old practice of
modular systems.

I am not sure how this might apply to large Ruby apps, my gut feeling
is there are not many huge ruby apps out there. The biggest I have
heard is around 30000 LOC, which is an awful lot for Ruby. Equivalent

I don't think it's that big.

functionality in a more conventional language (Java, C++, NET etc.)
would be bigger.

Rails apps are a bit different though, as Rails is an opinionated framework,
and logically dictates where your code should be. You also don't end up
writing much Ruby code. If you are, you're probably not leaning on the

I'm sure my use of rails is less than optimal. I still don't fully
undestand it.

stack enough. My gut feeling is that code bloat in Rails will very quickly
reveal itself that you are doing something wrong (possibly when test code
starts to get difficult to write). To get 30000 LOC in a Rails app, when it
is written well, your app would be huge, massively featured and have a
pretty huge UI (lots of RHTML), it might be possible to achieve that if you
are localising views a lot (while globalise nicely lets you avoid this as
much as possible, things like date controls, right->left reading order,
might make it easier to localise at the template level.

Thank you.
Hugh

 

[Hugh Sasse]

Invariant checks ("Design by Contract"), mostly. Adding invariant
checks to your system will often unearth weird cases - making them
easily reproducable and making them fail early.

Yes, this is something I need to know more about. In Programming Pearls
I've seen lots of mention of invariants, e.g. for loops, but though
I think I understood what was being said, I found that I didn't 'grok'
the statements deeply. I think I need to understand more about how to
think in this way. Some of this is probably covered by "What could
I put in an assert statement that would always succeed (for this loop
or whatever", but I suspect the concept runs deeper.

Testing (which has already been mentioned) can handle some of the same
niche, by removing coupling and allowing you to externally check for
invariants being kept. However, it is usually much easier to add
invariant checks to a system than to add tests.

Eivind.

Thank you,
Hugh

 

[Hugh Sasse]

A couple of questions:

1. How large is "large"? Is there some kind of "code size metric" that
the Ruby community uses, and a tool or tools to measure it?

Well, I'm trying to find out about techniques for use where
complexity and size are problem. What is large for me may not
be large for others. (I mean, there are some really sharp
programmers on this list!)

2. You say "your code". How much of this application have you personally
written, how much is "Rails and Ruby and the rest of the
infrastructure", and how much is "the rest of it"?

I have written all the stuff that is not rails or ruby libraries
out of stdlib. And it took me a months to write, but under
pressure, so it probably needs rewriting in many places.[Parallel history of debugging vs other CS stuff?]

The "traditional CASE tools" -- IDEs, software configuration and project
management tool sets, the waterfall model, the CMM levels, and of course
prayer, threats, outsourcing and pizza.

I clearly need to find out what IDEs exist for Ruby under
Unix/Cygwin, and what they'll do for me. I mainly use vim,
but then I suspect that I'm "so amazingly primative that
still think [syntax highlighting is] a pretty neat idea". I will
need to be able to configure any such IDE for large print, and
light text on dark, which will be a factor in what I choose.

Again, without knowing both the scope of your specific project nor the
size of the team that built/is building it, it's difficult to answer.

I am the team. I'd love to be able to code-review this stuff with
others here, but there's only me. So if it won't fit in my head
then....

There are bazillions of failed silver bullets to choose from. My
personal opinion is that you're being too hard on yourself and that
anybody who claims to have a tool or a process or a programming language
that is *significantly* better than today's common practices is either
deceived, deceiving or both.

Yes, I've read this sort of thing in The Mythical Man Month, but I'm
wondering about all the tech I don't know about. For example, much
of XP seemed to be established practice that I didn't know about
until I found out about RUnit circa 2001. So since all these other
fields I mentioned have progressed, it seems more likely that stuff
has passed me by than that it doesn't exist yet. I don't understand
Aspect Oriented Programming yet, for example.

I think first of all, you have to *want* to debug large chunks of other
peoples' code. It's an acquired taste. I acquired it at one point in my

Some have it thrust upon them. I feel like someone with a spade,
who's thinking, "They've got steamrollers and cranes, there must
be mechanical diggers by now".

career but found it unsatisfying. If you *don't* want to debug large
chunks of other peoples' code, there are ways you can structure your
team and processes to minimize how much of it you have to do.

Is that, 'for large values of "team"', only?

And I would caution you that, although testing and test-driven
development are certainly important and worthwhile, testing can only
show the *presence* of defects, not the absence of defects.

I know these ones are present, all right!

At the point in my career when I was at the peak of my ability to debug
other peoples' code, I came up with a simple rule. You'll probably need
to adjust the time scales to suit your situation, but in my case, the
rule was: If I don't find the problem in one day, it's not my mistake,
but a mistake in someone else's work. And if it takes more than a week,
it's not a software problem, it's a hardware problem.

I probably need to use the tools more effectively for this case, to
reach those values.

Good luck ... may the source be with you.

Thank you,
Hugh

 

[Hugh Sasse]

My first bet would be on improving the test case suite. But in the
process I think that a good debugger is a very valuable tool too. You
can try ruby-debug, which I find very helpful sometimes:

http://datanoise.com/articles/2006/09/14/debugging-rails-application

Thank you for this. That will help considerably

 

[M. Edward (Ed) Borasky]

I clearly need to find out what IDEs exist for Ruby under
Unix/Cygwin, and what they'll do for me. I mainly use vim,
but then I suspect that I'm "so amazingly primative that
still think [syntax highlighting is] a pretty neat idea". I will
need to be able to configure any such IDE for large print, and
light text on dark, which will be a factor in what I choose.

On Linux, and probably other UNIX-like operating systems, FreeRide is
available. I don't know if FreeRide will run under CygWin, but the
Windows One-Click installer contains a native Windows version of
FreeRide. Most of the folks here who run Ruby on Windows don't like the
CygWin version of Ruby, although if you want to run C extensions, it's
probably still a bit more convenient. In any event, I've had pretty good
luck mixing native Windows tools with CygWin -- if there's an equivalent
in CygWin, you usually have to do things with your path, but that's
about it.

I am the team. I'd love to be able to code-review this stuff with
others here, but there's only me. So if it won't fit in my head
then....

Well ... knowledge is meant to be shared ... teach someone Rails .

Yes, I've read this sort of thing in The Mythical Man Month, but I'm
wondering about all the tech I don't know about. For example, much
of XP seemed to be established practice that I didn't know about
until I found out about RUnit circa 2001. So since all these other
fields I mentioned have progressed, it seems more likely that stuff
has passed me by than that it doesn't exist yet. I don't understand
Aspect Oriented Programming yet, for example.

Or generative programming, or domain engineering, or domain specific
languages or language workbenches. Oh, yeah ... design by contract ...
proving programs correct ... the list is endless.

I'll claim (again) that the two most important factors in programmer
productivity are detailed knowledge of the application domain and
*familiarity* with the syntax and semantics of the programming language
and other development tools. A *bad* environment can destroy
productivity, but a good one doesn't guarantee it -- you need an
acceptable to good environment and detailed familiarity with it.

Some have it thrust upon them. I feel like someone with a spade,
who's thinking, "They've got steamrollers and cranes, there must
be mechanical diggers by now".

I suspect an IDE is the closest to a mechanical digger.

Is that, 'for large values of "team"', only?

Well ... more like large values of budget, I think.

I probably need to use the tools more effectively for this case, to
reach those values.

In my case, the product in question was a smallish operating system
which shall remain nameless. The tools were an assembler, documentation
of the OS table structure and core dumps. There was in fact a "kernel
debugger" but I rarely used it -- it was too risky on a live system. And
that was after having been to a training on the OS from the vendor and
two years of immersion in it at the assembly language level (this was BC
-- Before "C").

 

[Robert Klemme]

Yes, this is something I need to know more about. In Programming
Pearls I've seen lots of mention of invariants, e.g. for loops, but
though
I think I understood what was being said, I found that I didn't 'grok'
the statements deeply. I think I need to understand more about how to
think in this way. Some of this is probably covered by "What could
I put in an assert statement that would always succeed (for this loop
or whatever", but I suspect the concept runs deeper.

I found Bertrand Meyer's "OO Software Construction" provided a lot
interesting insights in this area and others as well. It's a large book and
it also covers Eiffel "along the way" but BM definitively has strong
opinions and well thought out positions that are worthwhile to consider -
even if you do not agree to all of them. I found the read very fruitful.

Kind regards

robert

 

[Robert Klemme]

I've delayed replying for a bit because lots of good stuff is
coming in, and I didn't want to stem the flow or divert it, ...



I'm not yet clear where to set the breakpoints, and so on, because
I don't understand enough about the nature of the failure. But
I probably need to re-examine the Ruby debugger, because my knowledge
of it's capabilities is somewhat limited.... So, up to now, no,
I have avoided this approach.

Maybe set_trace_func with a complete application trace (of the part that
fails) helps in understanding what methods get called. Maybe you then
realize immediately where the bug is (some method called too often or not at
all). If not you might get an idea where to place your breakpoints or
detailed debug output.

Well, the logs show the save! commands are getting called OK, with
the right values. I've said I've no reason to suspect the database.
Leaving reason aside, my emotions tell me that maybe I should
suspect, having read rather uncomplimentary stuff about MySQL,
advocating PostgreSQL, but I don't know enough about the DB issues
to be sure. This is my first serious DB project. So for now I
suspect my own code. So I'll need to refactor to improve testing.

Just some questions into the blue: are there triggers involved on DB level?
Do you use stored procedures vs. simple INSERT, UPDATE, DELETE, SELECT?

Kind regards

robert

 

[Robert Klemme]

The "traditional CASE tools" -- IDEs, software configuration and
project management tool sets, the waterfall model, the CMM levels,
and of course prayer, threats, outsourcing and pizza.

Coffee!
;-)

robert

 

[Hugh Sasse]

I found Bertrand Meyer's "OO Software Construction" provided a lot interesting
insights in this area and others as well. It's a large book and it also
covers Eiffel "along the way" but BM definitively has strong opinions and well
thought out positions that are worthwhile to consider - even if you do not
agree to all of them. I found the read very fruitful.

Thank you. I have a copy of that but have only dipped into it so far.

Kind regards

robert

Hugh

 

[Hugh Sasse]

Maybe set_trace_func with a complete application trace (of the part that
fails) helps in understanding what methods get called. Maybe you then realize
immediately where the bug is (some method called too often or not at all). If
not you might get an idea where to place your breakpoints or detailed debug
output.

That's a good idea, thank you.

Just some questions into the blue: are there triggers involved on DB level? Do
you use stored procedures vs. simple INSERT, UPDATE, DELETE, SELECT?

I'm using the ActiveRecord API to do all this find() and save() and save!().
Since the application is not concurrent at the moment I'm not doing anything
explicit with transactions, and I'm not using stored procedures either.
I'm not sure when to use stored procedures, for that matter.

Kind regards

robert

Thank you,
Hugh