Opinions on Java test development

[Michelle]

I am looking for some expert opinions. We are developing new Java
tests for a leading global consulting firm. The tests will assess the
skills of J2EE architects and Senior Java Developers. I could use
input on the differences in how to best assess various level
candidates. We will be looking for members of the test development
team once the focus of the test is finalized.

ReviewNet has been a leading provider of IT skills testing since 1997
and we have developed over 300 tests. We are determined to make this
the best Java suite of assessments possible and would like to get a
variety of input to make sure we are not biased in how we develop the
assessment. I look forward to hearing from you. I can be reached at
(e-mail address removed)

Thanks,

Michelle D. Bartie
ReviewNet Corporation
www.reviewnet.net
http://expert.reviewnet.net
phone: 215-572-9400 ext. 113

Rated Best by Inc. Magazine, and now with over 800 assessments for IT,
office skills, call center, and more!
http://www.inc.com/magazine/20001201/21124.html

 

[Roedy Green]

I could use
input on the differences in how to best assess various level
candidates. We will be looking for members of the test development
team once the focus of the test is finalized.

If I were devising a test, it would be an open book exam. The student
would even be allowed to use google and the newsgroups, just like real
life. Why make the test environment any more artificial than it
absolutely has to be?

You give him X hours to create a working miniature system from specs,
perhaps involving some new technology that no student would have ever
heard of.

For more advanced students you have him write the spec too based on
wooly customer needs. What a high level person has to be able to do
is come up with something plausible then get feedback on it rather
than relying on the customer to articulate his needs.

All you really can do with multiple choice is measure familiarity.

 

[Thomas Hawtin]

You give him X hours to create a working miniature system from specs,
perhaps involving some new technology that no student would have ever
heard of.

I think toy problems are fundamentally different from serious systems.

IMO, commenting on code is a far better way to go. Although that isn't
amenable to an automated test, so doesn't fit what this company wants to do.

Tom Hawtin

 

[Roedy Green]

IMO, commenting on code is a far better way to go. Although that isn't
amenable to an automated test, so doesn't fit what this company wants to do.

I think what the company want to do is foolish. If they truly want to
test somebody they should give a test that means something, not that
is easy to administer. A new hiree is not all that useful during the
first year. After that, they really start to carry weight. You are
training them and familiarising them to a great extent to start. Since
you want someone who will hang around for the long haul, it pays to
measure him properly.

Note the ads. Companies are desperate to shorten that time, wanting
their hirees to be totally familiar with every tool they will use on
the job.

 

[Roedy Green]

I could use
input on the differences in how to best assess various level
candidates.

What you might be able to do successfully with multiple choice is a
bullshit detector.

Let's say someone said they had enveloped large scale EJB
applications.

What you need are questions that anyone who has DONE that would HAVE
to know, not arcane stuff, but that someone who had merely read a book
ABOUT EJB would not know. These can be very bread and butter things,
like "Which of these three queries is of the sort most commonly used
in EJB applications?"

 

[Thomas Hawtin]

What you might be able to do successfully with multiple choice is a
bullshit detector.

With, at a guess, 90% of programmers being pretty much worthless, a
bullshit detector has to be worth its weight in brass.

A good proportion of companies don't even seem to bother to test at all.
OTOH, they wont have anyone capable of usefully judging candidates anyway.

Let's say someone said they had enveloped large scale EJB
applications.

What you need are questions that anyone who has DONE that would HAVE
to know, not arcane stuff, but that someone who had merely read a book
ABOUT EJB would not know. These can be very bread and butter things,
like "Which of these three queries is of the sort most commonly used
in EJB applications?"

That gets you as far as the one month experience level (possibly
repeated dozens of times). It fails to tell you whether the candidate is
competent. While certainly not foolproof, awareness of the obscure
indicates that the candidate has experience gained through thinking
through and solving problems. If someone has lots of time served, yet
haven't been aware of what's going on, then they're idiots.

Tom Hawtin

 

[Thomas Hawtin]

I think what the company want to do is foolish. If they truly want to
test somebody they should give a test that means something, not that
is easy to administer. A new hiree is not all that useful during the
first year. After that, they really start to carry weight. You are
training them and familiarising them to a great extent to start. Since
you want someone who will hang around for the long haul, it pays to
measure him properly.

Or her.

If a new hire takes a year to pull their weight, then I think you have
problems. Certainly there is going to be a bit of time (months) when not
only is the new employee not productive, but they reduce the
productivity of others. The critical factor in getting up to speed is
the ability to learn and think for yourself.

Note the ads. Companies are desperate to shorten that time, wanting
their hirees to be totally familiar with every tool they will use on
the job.

That's more to do with laziness, incompetence and ego boosting. My first
job had loads of languages listed. Apparently it's not uncommon for EJB
to be an absolute requirement where EJB is not used, has not been used
and there are no plans for its use. Listing packages rather than
requiring abilities has got to be an inferior technique to actually
testing something (or anything!).

Tom Hawtin

 

[Monique Y. Mudama]

I think what the company want to do is foolish. If they truly want
to test somebody they should give a test that means something, not
that is easy to administer.

I agree.

This may not be at all what you're thinking of, but ...

A job I applied for several months ago required applicants to do an
online Java language test. It was timed, but they didn't tell you it
was timed until after you started the test. It didn't allow you to
use reference materials, which doesn't represent any real-world
development environment I can think of. And the questions ...

The questions were basically of the form "Here is some code that no
one in their right mind would ever write. It uses obscure syntax,
requires intimate understanding of precedence rules, etc etc and
anyone who codes like this should immediately be taken out back and
shot. Describe what happens when it's run."

Certainly this test will weed out a lot of people. But will it
identify the types of developers you really want on your team? I
guess I find that doubtful.

I'll admit that I did poorly on the test. So maybe I'm biased.

On the other hand, when I did technical interviews with real live
people, the questions were a lot better at reflecting real world
needs. Things like explaining scenarios in which you would use
threads, inheritance, etc.

 

[Thomas Hawtin]

The questions were basically of the form "Here is some code that no
one in their right mind would ever write. It uses obscure syntax,
requires intimate understanding of precedence rules, etc etc and
anyone who codes like this should immediately be taken out back and
shot. Describe what happens when it's run."

I put some similar comments on the SCJP 5.0 beta.

Most code looks as if it was written by some who should be forcibly
restrained from going anywhere near a keyboard.

On the other hand, when I did technical interviews with real live
people, the questions were a lot better at reflecting real world
needs. Things like explaining scenarios in which you would use
threads, inheritance, etc.

Most technical interviewers are pants. It's rare to get asked questions
by someone with a clue. Thinking back I can only remember four that were
worthwhile. The two best of which were with, I believe, C++ programmers.

Tom Hawtin

 

[Roedy Green]

The questions were basically of the form "Here is some code that no
one in their right mind would ever write. It uses obscure syntax,
requires intimate understanding of precedence rules, etc etc and
anyone who codes like this should immediately be taken out back and
shot. Describe what happens when it's run."

Sometimes, unfortunately, this is a VERY realistic test.

There is quite a different skill set required to write code from
scratch and to make changes to a massive program written by someone
else.

 

[Michelle]

Thanks to everyone that commented on my post. I agree that there can
be some serious problems with on-line testing. That is we we are
looking to take a different approach. We may need to utilize a
different format other than mutiple choice. Thanks again for your
feedback!

Michelle

 

[Monique Y. Mudama]

Most technical interviewers are pants. It's rare to get asked
questions by someone with a clue. Thinking back I can only remember
four that were worthwhile. The two best of which were with, I
believe, C++ programmers.

I'm not familiary with the expression "are pants." What does that
mean?

Maybe you and I disagree on what constitutes a good technical
interview, but to me a general understanding of programming and
engineering principles and methodologies is more important than being
able to describe obscure points of a specific language.

 

[Monique Y. Mudama]

Sometimes, unfortunately, this is a VERY realistic test.

There is quite a different skill set required to write code from
scratch and to make changes to a massive program written by someone
else.

I've only rarely been able to work on a project that involved only my
code. I still think that the ability to understand purposely obtuse
code is only a small part of the skill set you need to work in that
kind of environment. From my perspective, I'd rather hire people who
will rewrite that code to make it readable (and still do the same
thing) than people who find it relatively easy to follow the
disgusting code (and hence don't see anything wrong with it).

 

[Thomas Fritsch]

I'm not familiary with the expression "are pants." What does that
mean?

According to my dictionary "pants" is british slang for "utterly bad".

 

[Monique Y. Mudama]

According to my dictionary "pants" is british slang for "utterly
bad".

Any explanation of the etymology?

 

[Thomas Hawtin]

Maybe you and I disagree on what constitutes a good technical
interview, but to me a general understanding of programming and
engineering principles and methodologies is more important than being
able to describe obscure points of a specific language.

I don't think that the major thrust of an interview should be obscure
points. However, if the candidate knows the obscure that is probably
because they have a very good general understanding of the subject and
can think. You will find it extremely difficult to find someone who
knows the obscure but doesn't have a good general understand of
programming and engineering principles. Similarly someone who has
developed a very detailed understanding of a system will certainly have
come across many of the obscurities.

As a side point, obscurities are not necessarily useless, although they
may be. Many form traps. If you are successfully going to code review,
then to complete the task you need to be sure it does exactly what you
expect (not a sufficient skill, but a necessary one). If you are going
to solve a problem (banging out the trivial is irrelevant), then you
need to understand the bounds and not be afraid to determine what is
possible and what is not.

Tom Hawtin

 

[Thomas Fritsch]

Any explanation of the etymology?

<http://www.ocf.berkeley.edu/~wrader/slang/p.html> says this:
inadequate, displeasing, or of poor quality. Possible origin: underwear,
called "pants" in Britain.

 

[Scott Ellsworth]

I don't think that the major thrust of an interview should be obscure
points. However, if the candidate knows the obscure that is probably
because they have a very good general understanding of the subject and
can think. You will find it extremely difficult to find someone who
knows the obscure but doesn't have a good general understand of
programming and engineering principles. Similarly someone who has
developed a very detailed understanding of a system will certainly have
come across many of the obscurities.

Actually, I have several counterexamples I have met - people who studied
for a cert test, and thus know a whole bunch of trivia, without any real
knowledge of how these features would be used. People who could tell me
all sorts of cruft about exactly how log4j implements a logger, without
being able to tell me a coherent logging policy.

Similarly, when pressed on how they would evaluate and adopt a new
technology, they tend to founder. They learned their trivia from a few
sources, rather than finding out about the trivia during the course of
implementing something important.

I really try to test prospective hires on their knowledge of the skills
I need, and their ability to learn the new stuff I do not yet know I
need. This is nontrivial - you have to make a judgement call.

I might start this process with a block of really cruddy code, then ask
how they would approach fixing it. Some trivia might be nice, but the
key is how they think.

Scott

 

[Roedy Green]

You will find it extremely difficult to find someone who
knows the obscure but doesn't have a good general understand of
programming and engineering principles.

Yes, but on the other hand, you may find that person a ditherer who
can't get on with real work. He is always fiddling with things that
are not that important. If he knows this stuff, it is likely because
is always pushing the envelope. This may or may not be the sort of guy
you want. If you want him maintaining applications, you don't want an
envelope-pusher guy. You want someone happy to write very mainstream
pedestrian code.

 

[Thomas Hawtin]

Yes, but on the other hand, you may find that person a ditherer who
can't get on with real work. He is always fiddling with things that
are not that important. If he knows this stuff, it is likely because
is always pushing the envelope. This may or may not be the sort of guy
you want. If you want him maintaining applications, you don't want an
envelope-pusher guy. You want someone happy to write very mainstream
pedestrian code.

That's a variation on the gold-plating myth. It's a popular one with
managers who see a mock-up but don't understand that nothing works. I
understand it doesn't really happen in real life. Unless you count
witting tests and the such like.

So, you'd go for pedestrian, copy & paste guy?

Suppose you were developing a product and it was decided that testing
should be written down as steps to follow in Word documents with visual
checking of the results. Would you prefer the engineers going for that,
or the one who spends an hour or two adapting the automated testing
framework?

The people you have to look out for is those who use inappropriate
"technology" to put on their CV. Those people who just have to use OLE
or EJB or whatever.

Tom Hawtin