software requirements again, take 483

[ccc31807]

We have an institutionally critical process that consists of a back
end (getting data from various sources and creating input files to
generate the documents) and a front end (preparing and sending the
documents from the input files.) We do this multiple times a year, and
it's task based, with single individuals performing multiple tasks
(from one to about six), and it generates hundreds of task documents
institution wide. Several years ago, I wrote the back end, and it's
worked perfectly.

A month or so ago, I was given the front end process as well. The due
date was Tuesday a week ago, nine days as I write. It was a week late,
I finished it last Tuesday, a week late. (I had to wait on input from
other people to finish the job, and they were tardy getting back to
me.) About 3:00 p.m., I got a call from the manager in charge who told
me that the software wasn't doing some of the documents correctly,
some weren't prepared at all and some were prepared wrong. Four
documents were affected out of 489.

After looking at it, I saw that the problem was the some tasks could
also be performed by multiple individuals, and my scripts only allowed
for tasks to be performed by one person. (The main database we use
contains a 'task' table, with one row per task and the individual
responsible for the task. It doesn't not contain information about
multiple individuals per task.) To correct the problem, I literally
had to start from the very beginning -- the first problem is that the
data now has to come from two tables that do not share a common key --
and revise the database queries, data formats, data structures, etc.

I griped (to anyone who would listen) that this requirement to
generate task documents accounting for tasks to be performed by
multiple individuals was a critical functional requirement, and I
DIDN'T FIND OUT ABOUT IT UNTIL THE SOFTWARE WAS A WEEK LATE!
It wasn't that this requirement wasn't known, these documents have
been generated for years, the manager wasn't aware of the details, it
just worked, and assumed that I knew the details. The guy who did the
front end had his methods for accounting for this requirement, but he
wasn't available. I'm guessing that he made the corrections manually,
but he had access to sources of data that I do not have.

It mystifies me that a critical requirement of this nature wouldn't be
specified until after the software had been written, and when the
software was already late. Apparently there's some universal law that
states this, but I don't know what the law is, or why the law is.

I know what some on c.l.p.m. will say -- don't correct the software
but comply with the wrong requirements, or get another job, but these
aren't realistic.

I'm just blowing off steam, decompressing. I've been pretty beat up
over the last three weeks, I'm about to take the rest of the week off,
and I just wanted to relate my experience. Thanks for listening.

CC.

 

[Charlton Wilbur]

cc> It mystifies me that a critical requirement of this nature
cc> wouldn't be specified until after the software had been written,
cc> and when the software was already late. Apparently there's some
cc> universal law that states this, but I don't know what the law
cc> is, or why the law is.

This is not mystifying. It has to do with the people who do the job
every day not thinking about the details, because they're internalized.

The problem is that this lack of communication has somehow become *your*
problem, and you seem to have accepted that. Doing that just sets
yourself up for more frustration and more failure.

Charlton

 

[Jürgen Exner]

DIDN'T FIND OUT ABOUT IT UNTIL THE SOFTWARE WAS A WEEK LATE!
It wasn't that this requirement wasn't known, these documents have
been generated for years, the manager wasn't aware of the details, it
just worked, and assumed that I knew the details. The guy who did the
front end had his methods for accounting for this requirement, but he
wasn't available. I'm guessing that he made the corrections manually,
but he had access to sources of data that I do not have.

It mystifies me that a critical requirement of this nature wouldn't be
specified until after the software had been written, and when the
software was already late. Apparently there's some universal law that
states this, but I don't know what the law is, or why the law is.

So, what was the feedback from those people when you presented your
spec? Did they miss the flaw, too, and signed off on the misguided
concept?
And what about during the testing phase? Yes, testing should primarily
check if the code complies with the written specification, but at least
in the later parts when actual users check out if the progam meets their
needs, then they should have noticed that critical functionality is
missing.


Honestly, looking at your long history here in this NG you are in this
way over your head and while you are trying to make up with a lot of
enthusiasm and determination it is hard to overlook that you are missing
basic skills for the job you are supposed to do.

It's like you know how to use a hammer and a saw and you are actually
rather skillful with those. So you are building garden sheds and the
occasional deck or gazebo and you turn them out on demand, in decend
quality and functional. But that doesn't automatically qualify you to
build a house or like in this case an add-on garage with mother-in-law.

Those projects simply require planning, organization, and preparation
skills (aka project management) which you don't have. And, if you excuse
me for saying so, which you have shown little interest to learn or to
adapt.
It's really the same story over and over again: you need to get your
nose out of the editor and stop worrying about those pesky syntax
errors. The actual coding part is only one small portion of the whole
development process and you have to start looking at the bigger picture
and understand development processes in context or things like this here
will keep happening again and again and every single time you will get
more and more frustrated.

Now, I do understand that there are people who couldn't care less about
project management and love spending all their day coding from dawn to
dusk instead. Fair enough, there's nothing wrong with that.
But based on your stories your position is not a position where you have
the luxury to do that. There is nobody else who would provide this
service for you, nobody you would talk with the users, gather and
organize their requirements, organize your coding work, organize the
schedule, testing, feedback gathering, and and and.
Therefore you either have to grow into that role or find someone else to
cover that role. Because as long as you keep applying the same
non-functioning process you cannot expect the outcome to change for the
better.

jue

 

[ccc31807]

The problem is that this lack of communication has somehow become *your*
problem, and you seem to have accepted that.  Doing that just sets
yourself up for more frustration and more failure.

I agree that the problem is lack of communication. As to the back end,
my main database query used a table that had EXACTLY one person per
task. As I mentioned, I have been doing this for years, and had ALWAYS
submitted a data file with one person per task.

The intermediate person, the one who is not available to talk to,
apparently manually adjusted the data files, because the ultimate
file, when delivered to the manager, was correct. In this case, the
requirement that on a few occasions a multiple number of individuals
could be assigned one task was 'unknown' in the sense that I didn't
know it, and the managing executive didn't know it.

I agree that the software had a bug, and the bug was my
responsibility, but how do you protect yourself from 'unknown'
requirements, in the absence of the person who actually KNOWS the
requirements?

CC

 

[Charlton Wilbur]

cc> I agree that the software had a bug, and the bug was my
cc> responsibility, but how do you protect yourself from 'unknown'
cc> requirements, in the absence of the person who actually KNOWS
cc> the requirements?

There are dozens of books written on this, and understanding their
contents is what separates a software engineer from a code monkey.

The fundamental rule, however, is that before you type a single
character of Perl you should have a clear, unambiguous statement, agreed
to by the business owner of the process you're automating or
facilitating, of what the requirements are for the phase of development
that you're in. When you have that statement, then you can estimate how
long it will take and what it depends on. Everything else is details.

Your problem, professionally, is that you don't get that unambiguous
statement, and if you do, and it turns out to be wrong, you accept
the blame and vent here. So long as you accept the blame and fix it,
the people in your organization have no incentive to change. If they
get the requirements wrong, they just pass the blame on to you. As long
as you keep on accepting it, nothing will change.

Your responsibility is to make sure you understand as well as possible
what the business process owner is asking for, and to give him what he
asks for. His responsibility is to ask for what he really wants. There
is no reasonable way for you to take on his responsibility, and your
biggest problem is that you allow him to foist it off on you.

Charlton

 

[Jim Gibson]

I agree that the problem is lack of communication. As to the back end,
my main database query used a table that had EXACTLY one person per
task. As I mentioned, I have been doing this for years, and had ALWAYS
submitted a data file with one person per task.

The intermediate person, the one who is not available to talk to,
apparently manually adjusted the data files, because the ultimate
file, when delivered to the manager, was correct. In this case, the
requirement that on a few occasions a multiple number of individuals
could be assigned one task was 'unknown' in the sense that I didn't
know it, and the managing executive didn't know it.

I agree that the software had a bug, and the bug was my
responsibility, but how do you protect yourself from 'unknown'
requirements, in the absence of the person who actually KNOWS the
requirements?

You get a set of written requirements and implement your program to
comply with them. You will probably have to write the requirements
yourself, since nobody else will think it is a worthwhile exercise. So
you do it, then get sign off from your boss and the customer
(end-user).

Then, when the program fails to meet the end-user's expectations, you
point out how the program actually complies with the written
requirements to which they agreed. Then, you revise the requirements to
match the user's expectations, modify the program to match the new
requirements, and move on to the next problem, iterating until
everybody is happy.

It is not really a software bug if the program complies with its
requirements. It is a requirements bug if the software complies with
the requirements but does not meet the user's expectations.

 

[Jürgen Exner]

I agree that the software had a bug, and the bug was my
responsibility,

No, it hadn't and no you weren't. The software behaved according to the
spec, so there was no bug in the software.

but how do you protect yourself from 'unknown'
requirements, in the absence of the person who actually KNOWS the
requirements?

By making sure that everyone who has a stake in the software attends the
spec review and signs off on the spec. If he doesn't speak up then a
flaw in the spec becomes _HIS_ problem, not yours.

jue

 

[Jürgen Exner]

cc> I agree that the software had a bug, and the bug was my
cc> responsibility, but how do you protect yourself from 'unknown'
cc> requirements, in the absence of the person who actually KNOWS
cc> the requirements?

There are dozens of books written on this, and understanding their
contents is what separates a software engineer from a code monkey.

The fundamental rule, however, is that before you type a single
character of Perl you should have a clear, unambiguous statement, agreed
to by the business owner of the process you're automating or
facilitating, of what the requirements are for the phase of development
that you're in. When you have that statement, then you can estimate how
long it will take and what it depends on. Everything else is details.

Your problem, professionally, is that you don't get that unambiguous
statement, and if you do, and it turns out to be wrong, you accept
the blame and vent here. So long as you accept the blame and fix it,
the people in your organization have no incentive to change. If they
get the requirements wrong, they just pass the blame on to you. As long
as you keep on accepting it, nothing will change.

Your responsibility is to make sure you understand as well as possible
what the business process owner is asking for, and to give him what he
asks for. His responsibility is to ask for what he really wants. There
is no reasonable way for you to take on his responsibility, and your
biggest problem is that you allow him to foist it off on you.

Amen to that, very well brought down to the point!

jue

 

[ccc31807]

Thanks for your comments. Please see below.

So, what was the feedback from those people when you presented your
spec?

There wasn't any spec. The organization changed the written
documentation, I was presented with the template, and was told to
'copy that.' Come to think of it, this probably WAS the specification.

Did they miss the flaw, too, and signed off on the misguided
concept?

The previous documents were one per task. The new documents are one
per person regardless of the number of tasks. I don't think anyone
realized that a task could be performed by multiple people -- not even
the big institutional database allowed it. The new format 'broke' the
old automation and thus had to be rewritten. (The new format was
dictated by HR, and the very first thing I brought up was why HR
didn't consult with IT as to the workability of the new format, but HR
doesn't have anything to do with IT.)

And what about during the testing phase? Yes, testing should primarily
check if the code complies with the written specification, but at least
in the later parts when actual users check out if the progam meets their
needs, then they should have noticed that critical functionality is
missing.

Actually, this was discovered during the testing phase. I had sent the
final set of documents to the manager and told her to examine them to
see if they had error, which they did. At least this part worked as
expected: test for errors, correct the errors, and test again. We
tested until there weren't any more errors.

Honestly, looking at your long history here in this NG you are in this
way over your head and while you are trying to make up with a lot of
enthusiasm and determination it is hard to overlook that you are missing
basic skills for the job you are supposed to do.

This is the truth. I'm not a programmer but a database guy. I get to
do programming because I am the only one who can, and I get to do it
all: analysis, design, implementation, testing and debugging,
deployment, and maintenance. It's a pretty good job, and I honestly
don't feel much stress.

It's like you know how to use a hammer and a saw and you are actually
rather skillful with those. So you are building garden sheds and the
occasional deck or gazebo and you turn them out on demand, in decend
quality and functional. But that doesn't automatically qualify you to
build a house or like in this case an add-on garage with mother-in-law.

That's right. This stretches me in terms of my skill level and
capabilities. I'm not complaining about that at all.

Those projects simply require planning, organization, and preparation
skills (aka project management) which you don't have. And, if you excuse
me for saying so, which you have shown little interest to learn or to
adapt.

That's at least partially true, but in mitigation, the vast majority
of my work consists of one off projects that literally can be
completed from start to finish in an hour or less, and this
environment does not inculcate the habit of planning, organization,
and preparation. Most things are so simple that you can start coding
from memory and never have to look at the Perl documentation or steal
code from past scripts.

It's really the same story over and over again: you need to get your
nose out of the editor and stop worrying about those pesky syntax
errors. The actual coding part is only one small portion of the whole
development process and you have to start looking at the bigger picture
and understand development processes in context or things like this here
will keep happening again and again and every single time you will get
more and more frustrated.

I may have mentioned that I'm in a solo ship, literally. I've not a
developer, or analyst, or even a programmer, but primarily a data
munger. I'm pretty good at what I do, and have a good reputation at
work for turning sow's ears into silk purses.

Now, I do understand that there are people who couldn't care less about
project management and love spending all their day coding from dawn to
dusk instead. Fair enough, there's nothing wrong with that.
But based on your stories your position is not a position where you have
the luxury to do that. There is nobody else who would provide this
service for you, nobody you would talk with the users, gather and
organize their requirements, organize your coding work, organize the
schedule, testing, feedback gathering, and and and.
Therefore you either have to grow into that role or find someone else to
cover that role. Because as long as you keep applying the same
non-functioning process you cannot expect the outcome to change for the
better.

All true. Again, in mitigation, my users sometimes don't know what
their requirements are. It's a common occurrence for me to furnish
information based on a written request that complies exactly with the
written request, and find out that the user really needs something
totally different. This happens maybe once in ten or twenty reports.

I like my job. The pay is lousy, but it's low stress, lots of off
time, not that much to do, I don't really have a boss that I have to
report to, I get time off to go to school, I get to play with
different technologies (including devoting a substantial time to
learning other languages, most recently Erlang and Lisp), and have the
respect of those who see what I do.

Plus, I enjoy griping, and you will accept my apology if I sometimes
favor c.l.p.m. with some of it.

CC.

 

[Peter J. Holzer]

You get a set of written requirements and implement your program to
comply with them. You will probably have to write the requirements
yourself, since nobody else will think it is a worthwhile exercise. So
you do it, then get sign off from your boss and the customer
(end-user).

Then, when the program fails to meet the end-user's expectations, you
point out how the program actually complies with the written
requirements to which they agreed. Then, you revise the requirements to
match the user's expectations, modify the program to match the new
requirements, and move on to the next problem, iterating until
everybody is happy.

The tricky part here is to make everybody happy. That not only means
that the users get what they want in the end, you must also avoid
frustration during the process. I think some points which help are:

1) Make clear from the start that the process is iterative. If both
sides are aware that the spec isn't carved in stone but can and will
be amended then it is much easier to do this.
2) Try to avoid passing blame. It isn't important whether the customer
forgot to tell something or changed their mind or whether the
programmer didn't ask the right questions or misunderstood the specs.
The important thing is that a flaw has been detected and must be
fixed. So write it in the specs and move on. (yes, some people always
want a scapegoat for every problem, preferrably before they even know
that there is a problem - but if they don't have to fear they they
will be blamed they are also less likely to blame you)
3) Keep the iterations short. Firstly, that will enable everybody to
remember what was discussed the last time and secondly, it avoids
wasting much time by going far into the wrong direction.
4) Fix the most important problems first. That way you will quickly
converge on a version which is usable, although it may not be
perfect.
5) As soon as the product is usable, use it! Avoid the temptation to
make it perfect. It never will be.

hp

 

[ccc31807]

The tricky part here is to make everybody happy. That not only means
that the users get what they want in the end, you must also avoid
frustration during the process. I think some points which help are:

1) Make clear from the start that the process is iterative. If both
sides are aware that the spec isn't carved in stone but can and will
be amended then it is much easier to do this.

Good. I like to iterate very quickly, sometimes several a day, The
people I work with frequently expect that I'll have version A ready by
noon and version B ready by the end of the day. I can do this because
my jobs are usually very simple and short.

2) Try to avoid passing blame. It isn't important whether the customer
forgot to tell something or changed their mind or whether the
programmer didn't ask the right questions or misunderstood the specs.
The important thing is that a flaw has been detected and must be
fixed. So write it in the specs and move on. (yes, some people always
want a scapegoat for every problem, preferrably before they even know
that there is a problem - but if they don't have to fear they they
will be blamed they are also less likely to blame you)

Good. My approach to this is to accept all the blame, whether or not
it's my fault. I don't have a problem with accepting the blame, and
many times that blame is deserved. ;-)

3) Keep the iterations short. Firstly, that will enable everybody to
remember what was discussed the last time and secondly, it avoids
wasting much time by going far into the wrong direction.

Good. See above. When the software only takes several hours to write,
this is easy to do. I have a project that requires more than a day (or
two) about once a month.

4) Fix the most important problems first. That way you will quickly
converge on a version which is usable, although it may not be
perfect.

Sometimes this is difficult because you don't know what the 'most
important' problem is before you start. Sometimes it's obvious, but
sometimes it's that little regex that turns out to be the pivot point.
Also, in my situation, requirements sometimes change radically. The
project I did last month was totally different from the written
specification (and yes, for this one I did have a written
specification.)

5) As soon as the product is usable, use it! Avoid the temptation to
make it perfect. It never will be.

Good. In my case, this isn't a problem because the focus is on the end
product, the report or document set. My users care nothing about the
scripting, all they are looking at is the result, which makes it easy
to focus on the major goal.

To these points, I would add:

6) Expect the requirements to be wrong. There are more bugs in the
specification than in the code, typically.
7) Expect the data to be bad. Something the in the input will always
break the script.
8) Expect the job to change. Often the user doesn't know what he wants
until you produce what he thinks he wants.
9) Be on your toes and be aware of your environment. Bullets fly! It
doesn't matter if the one that hits you is deliberately aimed or is a
ricochet.
10) Have fun -- this is probably the most important.

Thanks, CC.

 

[Charlton Wilbur]

JG> You get a set of written requirements and implement your program
JG> to comply with them. You will probably have to write the
JG> requirements yourself, since nobody else will think it is a
JG> worthwhile exercise. So you do it, then get sign off from your
JG> boss and the customer (end-user).

JG> Then, when the program fails to meet the end-user's
JG> expectations, you point out how the program actually complies
JG> with the written requirements to which they agreed. Then, you
JG> revise the requirements to match the user's expectations, modify
JG> the program to match the new requirements, and move on to the
JG> next problem, iterating until everybody is happy.

The trick here is to make it not about blame but about improved
communication. The business owner needs to understand that you do not
know all the details of the job he needs the software to do. So you
have to frame the whole thing not as "It complies with the written
requirements, so it's YOUR FAULT it's not what you want," but as "I
don't know your job as well as you do, which is why it's really
important that I understand what you need the software to do."

So if you write the requirements, the business process owner approves of
them, and the software matches the requirements, the bug is not your
fault, and you should push back HARD when the business process owner
tries to make it your fault. If you do this well, you can start
teaching the business process owner how to communicate better with you,
and get across to him the importance of accurately communicating
requirements.

Charlton

 

[Charlton Wilbur]

cc> Good. I like to iterate very quickly, sometimes several a day,
cc> The people I work with frequently expect that I'll have version
cc> A ready by noon and version B ready by the end of the day. I can
cc> do this because my jobs are usually very simple and short.

Do your iterations include clear statements from the business process
owners of what the expectations are for that iteration? If what you
mean is that they complain and you fix it quickly, that's not an
iteration, that's training them that you are a puppet that dances on the
strings they pull.

cc> Good. My approach to this is to accept all the blame, whether or
cc> not it's my fault. I don't have a problem with accepting the
cc> blame, and many times that blame is deserved. ;-)

That is NOT HELPFUL, and is a large contributor to your problem.

cc> Good. See above. When the software only takes several hours to
cc> write, this is easy to do. I have a project that requires more
cc> than a day (or two) about once a month.

How long do the *requirements* take to write? How long does it take to
get approval of them? What process do you follow to make sure the
requirements are corret? You're thinking like a code monkey, not a
software engineer.

cc> Sometimes this is difficult because you don't know what the
cc> 'most important' problem is before you start. Sometimes it's
cc> obvious, but sometimes it's that little regex that turns out to
cc> be the pivot point. Also, in my situation, requirements
cc> sometimes change radically. The project I did last month was
cc> totally different from the written specification (and yes, for
cc> this one I did have a written specification.)

You don't know what the most important technical problem is, but that's
completely irrelevant. What problem is most important to the business
process owner? I can guarantee you that he doesn't care about that
little regex.

And if requirements change radically, schedules should change radically
too. I see it even in good organizations: the developer provides an
estimate based on the requirements, the requirements change drastically,
and then the business process owner attempts to hold the developer to
the original estimate. It's the developer's responsibility to stand up
and say, "No, that's crazy talk. You changed the requirements, so the
schedule will change."

Also, if you start work on a project without written requirements,
you're contributing to the problem. You need to get in the habit, even
when things seem crystal clear, of writing the expectations down in an
email and sending them to the business process owner, saying, "This is
my understanding of what you need. Please correct me if I'm wrong, or
let me know if you think I've captured your requirements correctly."

Charlton

 

[Peter J. Holzer]

Good. I like to iterate very quickly, sometimes several a day, The
people I work with frequently expect that I'll have version A ready by
noon and version B ready by the end of the day. I can do this because
my jobs are usually very simple and short.

I was thinking of longer projects. For something that can be done in
half a day, revising the specs (if there even is one) isn't a problem.
But if you have spent half a year writing the specs and six different
people have signed it, it becomes damn hard to change it, even if parts
of it were obviously wrong.

Good. My approach to this is to accept all the blame, whether or not
it's my fault. I don't have a problem with accepting the blame, and
many times that blame is deserved. ;-)

That's not good and definitely not what I meant. You should not accept
all the blame (that's frustrating for you (and boy, you do sound
frustrated at times) and it undermines your reputation in the company
("that Carter guy never gets anything right")). Instead you should try
to create a working environment where blame isn't an issue. If something
like your one-task-several-workers problem pops up, whoever notices the
problem should just be able to say "hey, the program doesn't handle
tasks assigned to multiple persons right now, but it needs to. Can we
change that?" and not have to think about who they can blame and why
they didn't notice this 3 weeks ago.

Good. See above. When the software only takes several hours to write,
this is easy to do. I have a project that requires more than a day (or
two) about once a month.

That point (and in fact most of the others) was aimed at larger
projects. I have unfortunately been involved in projects with iteration
times of several months, where from one meeting to the next nobody could
remember what was meant by that cryptic paragraph in the specs, or a lot
of work had to be scrapped because either the spec was wrong or too
vague (and nobody noticed until the part had been implemented).

Sometimes this is difficult because you don't know what the 'most
important' problem is before you start. Sometimes it's obvious, but
sometimes it's that little regex that turns out to be the pivot point.

The "little regex" is never the most important problem. The most
important problem is that a task can be assigned only to a single
person, or that the report cannot be imported into the accounting
program or that the distance between the logo and the first line is 2mm
too small.

Solving the problem may involve changing that little regex, but the
customer doesn't care about how you solve the problem, but only that you
do it and how long it takes (and how much it costs). So after
identifying the problems you determine a priority and solve the problems
in order of decreasing priority. And yes, it is quite possible that the
customer cares more about the placement of the logo than about about a
substantial problem that only affects only 4 out of 500 reports.

Also, in my situation, requirements sometimes change radically. The
project I did last month was totally different from the written
specification (and yes, for this one I did have a written
specification.)

See point 1) above.

Good. In my case, this isn't a problem because the focus is on the end
product, the report or document set. My users care nothing about the
scripting, all they are looking at is the result, which makes it easy
to focus on the major goal.

To these points, I would add:

6) Expect the requirements to be wrong. There are more bugs in the
specification than in the code, typically.
7) Expect the data to be bad. Something the in the input will always
break the script.
8) Expect the job to change. Often the user doesn't know what he wants
until you produce what he thinks he wants.

These are already covered by 1).

9) Be on your toes and be aware of your environment. Bullets fly! It
doesn't matter if the one that hits you is deliberately aimed or is a
ricochet.

This is related to 2). Try to keep people firing the bullets in the
first place. At least in your vicinity.

10) Have fun -- this is probably the most important.

That I can agree with.

hp

 

[ccc31807]

So if you write the requirements, the business process owner approves of
them, and the software matches the requirements, the bug is not your
fault, and you should push back HARD when the business process owner
tries to make it your fault.  If you do this well, you can start
teaching the business process owner how to communicate better with you,
and get across to him the importance of accurately communicating
requirements.

Charlton, yesterday (Friday) when I came to work, I found a new
project on my desk. It was a log file from our corporate call center.
The manager wanted to know the locations of the callers. He had
attempted to do some of it by hand, but that is an arduous task, and
he soon gave up.

It was pretty simple to do, and I did it, from scratch, in less than
four hours, including finding and parsing the telephone exchange data
(which was a little difficult). When I finished, I called the manager
in charge and asked what format he wanted the report in. The answer?
"I don't know."

Think about this -- a manager assigns a project without thinking it
through to the extent that he knew the format of the output. Here I
was, with nice work and a ton of data, and no way to return it.

This is a communication problem. It's not a failure to communicate,
but a failure to have something to communicate. It's difficult to say
something if you don't have anything to say.

We'll get this sorted out on Monday so I'm not worried about it. The
point is that the technical person can't work without knowing what the
requirements are, and at least in my situation, I am often assigned
work with unclear, ambiguous, or absent requirements.

Finally, maybe I should add that the whole process is iterative, and
that the client doesn't know how he wants the data until he sees the
data. In this case, the client and the developer work together through
a number of iterations until the client sees how he uses the data, and
the developer helps him my writing output in a number of formats.

(Quick story -- We had a big push for graphical formatting and
visualization, the bosses wanted to 'see' the data. I spent several
days and developed an application that took data and spat out gifs,
literally hundreds of gifs, bar charts, line graphs, etc. The final
result was a 40+ page PDF with eight charts on each page. Guess what?
After I submitted this, I never heard from it again. The theory about
visualizing data is good, but it has it's drawbacks as well, and you
might not realize that you really don't want to 'see' the data until
after you see it.)

CC

 

[Jürgen Exner]

Same old story again.

Charlton, yesterday (Friday) when I came to work, I found a new
project on my desk. It was a log file from our corporate call center.
The manager wanted to know the locations of the callers. He had
attempted to do some of it by hand, but that is an arduous task, and
he soon gave up.

It was pretty simple to do, and I did it, from scratch, in less than
four hours, including finding and parsing the telephone exchange data
(which was a little difficult).

Let me rephrase that for you: I jumped right into coding without
checking if the spec was accurate, the business guy and I understood it
the same way, or even if it was complete.

When I finished, I called the manager
in charge and asked what format he wanted the report in. The answer?
"I don't know."

Think about this -- a manager assigns a project without thinking it
through to the extent that he knew the format of the output.

And that is exactly what is biting you over and over again. The spec, no
matter if a 2 paragraph email or a 2000 page document, was incomplete
and you didn't even check before starting coding. It is not the job of
the manager but YOUR's to make sure that you have all the necessary
information to finish the task. That is a very basic software
engineering requirement.

Here I
was, with nice work and a ton of data, and no way to return it.

This is a communication problem. It's not a failure to communicate,
but a failure to have something to communicate. It's difficult to say
something if you don't have anything to say.

No, it is not a communication problem. It's a problem of poor processes.
You keep jumping to step 5 before completing or at the very least
thouroughly checking steps 2-4.

We'll get this sorted out on Monday so I'm not worried about it. The
point is that the technical person can't work without knowing what the
requirements are, and at least in my situation, I am often assigned
work with unclear, ambiguous, or absent requirements.

Then why on earth don't you ask for clarification? It is absolutely
essential and in the very own interest of a programmer to know exactly
what he is supposed to program before starting to program. And that's
what (a large part of) software engineering is all about: making sure
you know what you are supposed to do.
You cannot expect a business person or any user for that matter to
provide you with the perfect or even a meaningful specification.
Therefore it is YOUR(!) job to work with them, to discuss, to point out
open questions, and yes, even to help them to make up their mind how
certain things are supposed to look like or the output is supposed to be
formatted. And this doesn't have to be a 300 page spec written in blood
and notarized. A short email may be sufficient, but it absolutely needs
to be addressed and absolutely has to be in writing such that everyone
is working off the same expectations.
And that comes before(!!!) you write a single line of code.

Finally, maybe I should add that the whole process is iterative, and
that the client doesn't know how he wants the data until he sees the
data. In this case, the client and the developer work together through
a number of iterations until the client sees how he uses the data, and
the developer helps him my writing output in a number of formats.

Nothing wrong with that. If that is the preferred way, then it should be
stated as such in the spec. Then it becomes part of the process, is
expected, and can be scheduled and credited accordingly.

Without it being documented in the spec the conclusion will be: "Oh,
John just doesn't know what he is doing. I had to come back to him 5
times until finally he got the output right. And because of that we were
2 weeks late, too, and missed the deadline for the appliation."

(Quick story -- We had a big push for graphical formatting and
visualization, the bosses wanted to 'see' the data. I spent several
days and developed an application that took data and spat out gifs,
literally hundreds of gifs, bar charts, line graphs, etc. The final
result was a 40+ page PDF with eight charts on each page. Guess what?
After I submitted this, I never heard from it again. The theory about
visualizing data is good, but it has it's drawbacks as well, and you
might not realize that you really don't want to 'see' the data until
after you see it.)

The secret keyword for that is "prototyping". If he doesn't know exactly
what he is looking for (and that is not uncommon) then do a few
different prototypes for him to choose from. Then do a few more
variations with simple, limited data sets for those he favours. And only
after he confirms "This here, this is what I want", then you go full
steam ahead and implement the full solution.
Is it stands now you probably wasted several days of work while with
using prototypes either he would have gotten what he was looking for or
if charts were a bad idea to begin with he would have found out much
earlier after spending maybe just a few hours on the task.

jue

 

[ccc31807]

Let me rephrase that for you: I jumped right into coding without
checking if the spec was accurate, the business guy and I understood it
the same way, or even if it was complete.

The business guy was out of the office at a conference. All I had was
the log file and that he wanted it by Monday. Initially, I didn't even
know what it was that he wanted, but it wasn't difficult to guess. I
didn't want to wait until I knew the requirements in toto when I knew
I could do 90% with the information I had.

And that is exactly what is biting you over and over again. The spec, no
matter if a 2 paragraph email or a 2000 page document, was incomplete
and you didn't even check before starting coding. It is not the job of
the manager but YOUR's to make sure that you have all the necessary
information to finish the task. That is a very basic software
engineering requirement.

The 'spec' in this case is an email with no message, just an
attachment. After I figured out what he wanted, I had enough
information to do the heavy lifting. The format is immaterial to me as
I can stuff the data into any format he is likely to want (probably an
Excel file). I had all the NECESSARY information to basically complete
the job, but not enough to produce the final product.

No, it is not a communication problem. It's a problem of poor processes.

I totally agree.

Then why on earth don't you ask for clarification?

Because I couldn't. The clarifier was not available.

It is absolutely
essential and in the very own interest of a programmer to know exactly
what he is supposed to program before starting to program.

Disagree strongly. It's true that you need to know precisely what you
are doing before writing code, but particularly with agile processes,
you don't need the specs nailed 100% before starting to write some
code.

Nothing wrong with that. If that is the preferred way, then it should be
stated as such in the spec. Then it becomes part of the process, is
expected, and can be scheduled and credited accordingly.

Which is pretty much the way I work. People who work with me are
accustomed to have some back and forth before they get the final
product.

Without it being documented in the spec the conclusion will be: "Oh,
John just doesn't know what he is doing. I had to come back to him 5
times until finally he got the output right. And because of that we were
2 weeks late, too, and missed the deadline for the appliation."

John: "I'll take the blame for assuming you were not an idiot, and for
assuming that the previous requirements you gave me were accurate. I
should have known better." I have said this directly to some people,
and those people, while they don't care for me, do take pains not to
give me any excuse to say that their instructions were imprecise or
unclear.

The secret keyword for that is "prototyping". If he doesn't know exactly
what he is looking for (and that is not uncommon) then do a few
different prototypes for him to choose from. Then do a few more
variations with simple, limited data sets for those he favours. And only
after he confirms "This here, this is what I want", then you go full
steam ahead and implement the full solution.

That's essentially what I did with the project mentioned before. The
HR staff did not think through the process, and I had to go back to
them several times with a 'What now?' question. This was the reason
that the software was a week late -- their tardiness in answering my
questions. If these issues had been resolved in real time, I would
have finished on time. Also, it's stupid to blame the programmer for
lack of domain knowledge when the SMEs didn't have the appropriate
domain knowledge.

Is it stands now you probably wasted several days of work

Not true at all. I had been playing with GD for several years, wanting
to use it in code. This gave me the excuse to learn GD. I don't
consider that a waste, and in fact was able to leverage that
experience to complete a side job. In effect, I got paid twice for the
same work.

Ain't life grand.

CC.

 

[Peter J. Holzer]

[the project which started this thread took several weeks]

Charlton, yesterday (Friday) when I came to work, I found a new
project on my desk. It was a log file from our corporate call center.
The manager wanted to know the locations of the callers. He had
attempted to do some of it by hand, but that is an arduous task, and
he soon gave up.

It was pretty simple to do, and I did it, from scratch, in less than
four hours, including finding and parsing the telephone exchange data
(which was a little difficult). [...]
(Quick story -- We had a big push for graphical formatting and
visualization, the bosses wanted to 'see' the data. I spent several
days and developed an application that took data and spat out gifs,
literally hundreds of gifs, bar charts, line graphs, etc. The final
result was a 40+ page PDF with eight charts on each page. Guess what?

Carter, please do yourself and us a favour and try to distinguish
between projects which take a few hours and those which take days or
weeks (or months or years, but you don't seem to have those).

For something that takes a few hours I agree with you that it often
makes sense to knock up a prototype *before* asking the user for
clarification: It gives you a chance to look at the problem so that you
can ask the right questions, and it gives you something tangible to talk
about (some users have absolutely zero imagination). And if you know
your users there's a good chance that your prototype will already be
good enough.

However, for longer projects trying to solve 90% before asking the user
what they really want is just stupid. It's still a good idea to have a
look at the problem before asking for clarification so that you have an
idea what they are talking about and what's possible and that you can
ask the right questions. But before putting significant work into the
project you have to make that you do what the user wants (now - he will
probably change his mind later, but that's his problem).

I think Jürgen and you make the same mistake, but you come from
different directions:

You usually do small jobs (I actually hesitate to call them "projects")
where almost any formalization would cost time, not save it. And then
you try the same approach with larger projects (still rather small, but
somewhere beyound the size where a bit of planning pays) and are
frustrated that it doesn't work as well as for your small jobs.

Jürgen obviously comes from the opposite direction: Projects which have
a defined scope (probably in a business environment where the customer
and the contractor agree on a price beforehand and sign a contract). And
he doesn't see that there are jobs where working like this adds too
much overhead. You can't have a one hour meeting for a five minute job.

hp

 

[ccc31807]

I agree with almost everything below.

The only thing I would add is that planning is necessary for small
scripting tasks as well as the major projects, and that
experimentation and an ad hoc approach can yield insight into specific
areas in large projects as well as the smaller ones.

In a former life, I did a lot of extemporaneous speaking.
'Extemporaneous' doesn't means having no knowledge about a subject, it
actually requires a much deeper understanding of the subject that a
prepared speech. When called on to say say something (which likely
might be an extended analysis of a complex problem) it took be about
half a minute to develop an outline of the speech in my head while
make some introductory remarks (e.g., telling a joke or complimenting
the previous speaker).

Doing a small job is exactly the same. You can develop the pseudo code
in your head while writing:

#! usr/bin/perl
#informational header, input and output
use strict;
use warnings;
print "Running ad_hoc_job.plx\n"
exit(0);

The point is that some kinds of tasks become so familiar that you can
outline the script in your sleep. The problems tend to come in the
little regular expression that you need to break apart the input line
-- and THAT can take more time to write the the entire rest of the
job.

Thanks for your insightful comments, CC.

 

[Charlton Wilbur]

cc> We'll get this sorted out on Monday so I'm not worried about
cc> it. The point is that the technical person can't work without
cc> knowing what the requirements are, and at least in my situation,
cc> I am often assigned work with unclear, ambiguous, or absent
cc> requirements.

Yes, and when a professional software engineer is handed unclear,
ambiguous, or absent requirements, it is one of his responsibilities to
see that they are clarified, disambiguated, or written at all.

That is the point that you do not seem to wish to understand.

Charlton