Opinions on Java test development

Discussion in 'Java' started by Michelle, Sep 27, 2005.

  1. Michelle

    Michelle Guest

    I am looking for some expert opinions. We are developing new Java
    tests for a leading global consulting firm. The tests will assess the
    skills of J2EE architects and Senior Java Developers. I could use
    input on the differences in how to best assess various level
    candidates. We will be looking for members of the test development
    team once the focus of the test is finalized.

    ReviewNet has been a leading provider of IT skills testing since 1997
    and we have developed over 300 tests. We are determined to make this
    the best Java suite of assessments possible and would like to get a
    variety of input to make sure we are not biased in how we develop the
    assessment. I look forward to hearing from you. I can be reached at


    Thanks,

    Michelle D. Bartie
    ReviewNet Corporation
    www.reviewnet.net
    http://expert.reviewnet.net
    phone: 215-572-9400 ext. 113

    Rated Best by Inc. Magazine, and now with over 800 assessments for IT,
    office skills, call center, and more!
    http://www.inc.com/magazine/20001201/21124.html
    Michelle, Sep 27, 2005
    #1
    1. Advertising

  2. Michelle

    Roedy Green Guest

    On 27 Sep 2005 15:17:59 -0700, "Michelle" <>
    wrote or quoted :

    >I could use
    >input on the differences in how to best assess various level
    >candidates. We will be looking for members of the test development
    >team once the focus of the test is finalized.


    If I were devising a test, it would be an open book exam. The student
    would even be allowed to use google and the newsgroups, just like real
    life. Why make the test environment any more artificial than it
    absolutely has to be?

    You give him X hours to create a working miniature system from specs,
    perhaps involving some new technology that no student would have ever
    heard of.

    For more advanced students you have him write the spec too based on
    wooly customer needs. What a high level person has to be able to do
    is come up with something plausible then get feedback on it rather
    than relying on the customer to articulate his needs.

    All you really can do with multiple choice is measure familiarity.
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Again taking new Java programming contracts.
    Roedy Green, Sep 27, 2005
    #2
    1. Advertising

  3. Roedy Green wrote:
    >
    > You give him X hours to create a working miniature system from specs,
    > perhaps involving some new technology that no student would have ever
    > heard of.


    I think toy problems are fundamentally different from serious systems.

    IMO, commenting on code is a far better way to go. Although that isn't
    amenable to an automated test, so doesn't fit what this company wants to do.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 28, 2005
    #3
  4. Michelle

    Roedy Green Guest

    On Wed, 28 Sep 2005 00:22:54 +0100, Thomas Hawtin
    <> wrote or quoted :

    >IMO, commenting on code is a far better way to go. Although that isn't
    >amenable to an automated test, so doesn't fit what this company wants to do.


    I think what the company want to do is foolish. If they truly want to
    test somebody they should give a test that means something, not that
    is easy to administer. A new hiree is not all that useful during the
    first year. After that, they really start to carry weight. You are
    training them and familiarising them to a great extent to start. Since
    you want someone who will hang around for the long haul, it pays to
    measure him properly.

    Note the ads. Companies are desperate to shorten that time, wanting
    their hirees to be totally familiar with every tool they will use on
    the job.

    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Again taking new Java programming contracts.
    Roedy Green, Sep 28, 2005
    #4
  5. Michelle

    Roedy Green Guest

    On 27 Sep 2005 15:17:59 -0700, "Michelle" <>
    wrote or quoted :

    >I could use
    >input on the differences in how to best assess various level
    >candidates.


    What you might be able to do successfully with multiple choice is a
    bullshit detector.

    Let's say someone said they had enveloped large scale EJB
    applications.

    What you need are questions that anyone who has DONE that would HAVE
    to know, not arcane stuff, but that someone who had merely read a book
    ABOUT EJB would not know. These can be very bread and butter things,
    like "Which of these three queries is of the sort most commonly used
    in EJB applications?"
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Again taking new Java programming contracts.
    Roedy Green, Sep 28, 2005
    #5
  6. Roedy Green wrote:
    > On 27 Sep 2005 15:17:59 -0700, "Michelle" <>
    > wrote or quoted :
    >
    >
    >>I could use
    >>input on the differences in how to best assess various level
    >>candidates.

    >
    >
    > What you might be able to do successfully with multiple choice is a
    > bullshit detector.


    With, at a guess, 90% of programmers being pretty much worthless, a
    bullshit detector has to be worth its weight in brass.

    A good proportion of companies don't even seem to bother to test at all.
    OTOH, they wont have anyone capable of usefully judging candidates anyway.

    > Let's say someone said they had enveloped large scale EJB
    > applications.
    >
    > What you need are questions that anyone who has DONE that would HAVE
    > to know, not arcane stuff, but that someone who had merely read a book
    > ABOUT EJB would not know. These can be very bread and butter things,
    > like "Which of these three queries is of the sort most commonly used
    > in EJB applications?"


    That gets you as far as the one month experience level (possibly
    repeated dozens of times). It fails to tell you whether the candidate is
    competent. While certainly not foolproof, awareness of the obscure
    indicates that the candidate has experience gained through thinking
    through and solving problems. If someone has lots of time served, yet
    haven't been aware of what's going on, then they're idiots.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 28, 2005
    #6
  7. Roedy Green wrote:
    > On Wed, 28 Sep 2005 00:22:54 +0100, Thomas Hawtin
    > <> wrote or quoted :
    >
    >
    >>IMO, commenting on code is a far better way to go. Although that isn't
    >>amenable to an automated test, so doesn't fit what this company wants to do.

    >
    >
    > I think what the company want to do is foolish. If they truly want to
    > test somebody they should give a test that means something, not that
    > is easy to administer. A new hiree is not all that useful during the
    > first year. After that, they really start to carry weight. You are
    > training them and familiarising them to a great extent to start. Since
    > you want someone who will hang around for the long haul, it pays to
    > measure him properly.


    Or her.

    If a new hire takes a year to pull their weight, then I think you have
    problems. Certainly there is going to be a bit of time (months) when not
    only is the new employee not productive, but they reduce the
    productivity of others. The critical factor in getting up to speed is
    the ability to learn and think for yourself.

    > Note the ads. Companies are desperate to shorten that time, wanting
    > their hirees to be totally familiar with every tool they will use on
    > the job.


    That's more to do with laziness, incompetence and ego boosting. My first
    job had loads of languages listed. Apparently it's not uncommon for EJB
    to be an absolute requirement where EJB is not used, has not been used
    and there are no plans for its use. Listing packages rather than
    requiring abilities has got to be an inferior technique to actually
    testing something (or anything!).

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 28, 2005
    #7
  8. On 2005-09-28, Roedy Green penned:
    > On Wed, 28 Sep 2005 00:22:54 +0100, Thomas Hawtin
    ><> wrote or quoted :
    >
    >>IMO, commenting on code is a far better way to go. Although that
    >>isn't amenable to an automated test, so doesn't fit what this
    >>company wants to do.

    >
    > I think what the company want to do is foolish. If they truly want
    > to test somebody they should give a test that means something, not
    > that is easy to administer.


    I agree.

    This may not be at all what you're thinking of, but ...

    A job I applied for several months ago required applicants to do an
    online Java language test. It was timed, but they didn't tell you it
    was timed until after you started the test. It didn't allow you to
    use reference materials, which doesn't represent any real-world
    development environment I can think of. And the questions ...

    The questions were basically of the form "Here is some code that no
    one in their right mind would ever write. It uses obscure syntax,
    requires intimate understanding of precedence rules, etc etc and
    anyone who codes like this should immediately be taken out back and
    shot. Describe what happens when it's run."

    Certainly this test will weed out a lot of people. But will it
    identify the types of developers you really want on your team? I
    guess I find that doubtful.

    I'll admit that I did poorly on the test. So maybe I'm biased.

    On the other hand, when I did technical interviews with real live
    people, the questions were a lot better at reflecting real world
    needs. Things like explaining scenarios in which you would use
    threads, inheritance, etc.

    --
    monique

    Ask smart questions, get good answers:
    http://www.catb.org/~esr/faqs/smart-questions.html
    Monique Y. Mudama, Sep 28, 2005
    #8
  9. Monique Y. Mudama wrote:
    >
    > The questions were basically of the form "Here is some code that no
    > one in their right mind would ever write. It uses obscure syntax,
    > requires intimate understanding of precedence rules, etc etc and
    > anyone who codes like this should immediately be taken out back and
    > shot. Describe what happens when it's run."


    I put some similar comments on the SCJP 5.0 beta.

    Most code looks as if it was written by some who should be forcibly
    restrained from going anywhere near a keyboard.

    > On the other hand, when I did technical interviews with real live
    > people, the questions were a lot better at reflecting real world
    > needs. Things like explaining scenarios in which you would use
    > threads, inheritance, etc.


    Most technical interviewers are pants. It's rare to get asked questions
    by someone with a clue. Thinking back I can only remember four that were
    worthwhile. The two best of which were with, I believe, C++ programmers.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 28, 2005
    #9
  10. Michelle

    Roedy Green Guest

    On Wed, 28 Sep 2005 14:41:44 -0600, "Monique Y. Mudama"
    <> wrote or quoted :

    >The questions were basically of the form "Here is some code that no
    >one in their right mind would ever write. It uses obscure syntax,
    >requires intimate understanding of precedence rules, etc etc and
    >anyone who codes like this should immediately be taken out back and
    >shot. Describe what happens when it's run."


    Sometimes, unfortunately, this is a VERY realistic test. :)

    There is quite a different skill set required to write code from
    scratch and to make changes to a massive program written by someone
    else.
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Again taking new Java programming contracts.
    Roedy Green, Sep 28, 2005
    #10
  11. Michelle

    Michelle Guest

    Thanks to everyone that commented on my post. I agree that there can
    be some serious problems with on-line testing. That is we we are
    looking to take a different approach. We may need to utilize a
    different format other than mutiple choice. Thanks again for your
    feedback!

    Michelle
    Michelle, Sep 28, 2005
    #11
  12. On 2005-09-28, Thomas Hawtin penned:
    > Monique Y. Mudama wrote:
    >>
    >> On the other hand, when I did technical interviews with real live
    >> people, the questions were a lot better at reflecting real world
    >> needs. Things like explaining scenarios in which you would use
    >> threads, inheritance, etc.

    >
    > Most technical interviewers are pants. It's rare to get asked
    > questions by someone with a clue. Thinking back I can only remember
    > four that were worthwhile. The two best of which were with, I
    > believe, C++ programmers.
    >


    I'm not familiary with the expression "are pants." What does that
    mean?

    Maybe you and I disagree on what constitutes a good technical
    interview, but to me a general understanding of programming and
    engineering principles and methodologies is more important than being
    able to describe obscure points of a specific language.

    --
    monique

    Ask smart questions, get good answers:
    http://www.catb.org/~esr/faqs/smart-questions.html
    Monique Y. Mudama, Sep 29, 2005
    #12
  13. On 2005-09-28, Roedy Green penned:
    > On Wed, 28 Sep 2005 14:41:44 -0600, "Monique Y. Mudama"
    ><> wrote or quoted :
    >
    >>The questions were basically of the form "Here is some code that no
    >>one in their right mind would ever write. It uses obscure syntax,
    >>requires intimate understanding of precedence rules, etc etc and
    >>anyone who codes like this should immediately be taken out back and
    >>shot. Describe what happens when it's run."

    >
    > Sometimes, unfortunately, this is a VERY realistic test. :)
    >
    > There is quite a different skill set required to write code from
    > scratch and to make changes to a massive program written by someone
    > else.


    I've only rarely been able to work on a project that involved only my
    code. I still think that the ability to understand purposely obtuse
    code is only a small part of the skill set you need to work in that
    kind of environment. From my perspective, I'd rather hire people who
    will rewrite that code to make it readable (and still do the same
    thing) than people who find it relatively easy to follow the
    disgusting code (and hence don't see anything wrong with it).

    --
    monique

    Ask smart questions, get good answers:
    http://www.catb.org/~esr/faqs/smart-questions.html
    Monique Y. Mudama, Sep 29, 2005
    #13
  14. Monique Y. Mudama wrote:
    > On 2005-09-28, Thomas Hawtin penned:
    >
    >>Most technical interviewers are pants. It's rare to get asked
    >>questions by someone with a clue. Thinking back I can only remember
    >>four that were worthwhile. The two best of which were with, I
    >>believe, C++ programmers.

    >
    > I'm not familiary with the expression "are pants." What does that
    > mean?

    According to my dictionary "pants" is british slang for "utterly bad".

    --
    Thomas
    Thomas Fritsch, Sep 29, 2005
    #14
  15. On 2005-09-29, Thomas Fritsch penned:
    > Monique Y. Mudama wrote:
    >> On 2005-09-28, Thomas Hawtin penned:
    >>
    >>>Most technical interviewers are pants. It's rare to get asked
    >>>questions by someone with a clue. Thinking back I can only remember
    >>>four that were worthwhile. The two best of which were with, I
    >>>believe, C++ programmers.

    >>
    >> I'm not familiary with the expression "are pants." What does that
    >> mean?

    > According to my dictionary "pants" is british slang for "utterly
    > bad".


    Any explanation of the etymology?

    --
    monique

    Ask smart questions, get good answers:
    http://www.catb.org/~esr/faqs/smart-questions.html
    Monique Y. Mudama, Sep 29, 2005
    #15
  16. Monique Y. Mudama wrote:
    >
    > Maybe you and I disagree on what constitutes a good technical
    > interview, but to me a general understanding of programming and
    > engineering principles and methodologies is more important than being
    > able to describe obscure points of a specific language.


    I don't think that the major thrust of an interview should be obscure
    points. However, if the candidate knows the obscure that is probably
    because they have a very good general understanding of the subject and
    can think. You will find it extremely difficult to find someone who
    knows the obscure but doesn't have a good general understand of
    programming and engineering principles. Similarly someone who has
    developed a very detailed understanding of a system will certainly have
    come across many of the obscurities.

    As a side point, obscurities are not necessarily useless, although they
    may be. Many form traps. If you are successfully going to code review,
    then to complete the task you need to be sure it does exactly what you
    expect (not a sufficient skill, but a necessary one). If you are going
    to solve a problem (banging out the trivial is irrelevant), then you
    need to understand the bounds and not be afraid to determine what is
    possible and what is not.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 29, 2005
    #16
  17. "Monique Y. Mudama" <> wrote:
    > On 2005-09-29, Thomas Fritsch penned:
    >> Monique Y. Mudama wrote:
    >>> I'm not familiary with the expression "are pants." What does that
    >>> mean?

    >> According to my dictionary "pants" is british slang for "utterly
    >> bad".

    >
    > Any explanation of the etymology?
    >

    <http://www.ocf.berkeley.edu/~wrader/slang/p.html> says this:
    inadequate, displeasing, or of poor quality. Possible origin: underwear,
    called "pants" in Britain.

    --
    Thomas
    Thomas Fritsch, Sep 29, 2005
    #17
  18. In article <433c3e6c$0$73618$>,
    Thomas Hawtin <> wrote:

    > Monique Y. Mudama wrote:
    > >
    > > Maybe you and I disagree on what constitutes a good technical
    > > interview, but to me a general understanding of programming and
    > > engineering principles and methodologies is more important than being
    > > able to describe obscure points of a specific language.

    >
    > I don't think that the major thrust of an interview should be obscure
    > points. However, if the candidate knows the obscure that is probably
    > because they have a very good general understanding of the subject and
    > can think. You will find it extremely difficult to find someone who
    > knows the obscure but doesn't have a good general understand of
    > programming and engineering principles. Similarly someone who has
    > developed a very detailed understanding of a system will certainly have
    > come across many of the obscurities.


    Actually, I have several counterexamples I have met - people who studied
    for a cert test, and thus know a whole bunch of trivia, without any real
    knowledge of how these features would be used. People who could tell me
    all sorts of cruft about exactly how log4j implements a logger, without
    being able to tell me a coherent logging policy.

    Similarly, when pressed on how they would evaluate and adopt a new
    technology, they tend to founder. They learned their trivia from a few
    sources, rather than finding out about the trivia during the course of
    implementing something important.

    I really try to test prospective hires on their knowledge of the skills
    I need, and their ability to learn the new stuff I do not yet know I
    need. This is nontrivial - you have to make a judgement call.

    I might start this process with a block of really cruddy code, then ask
    how they would approach fixing it. Some trivia might be nice, but the
    key is how they think.

    Scott

    --
    Scott Ellsworth

    Java and database consulting for the life sciences
    Scott Ellsworth, Sep 30, 2005
    #18
  19. Michelle

    Roedy Green Guest

    On Thu, 29 Sep 2005 20:19:51 +0100, Thomas Hawtin
    <> wrote or quoted :

    > You will find it extremely difficult to find someone who
    >knows the obscure but doesn't have a good general understand of
    >programming and engineering principles.


    Yes, but on the other hand, you may find that person a ditherer who
    can't get on with real work. He is always fiddling with things that
    are not that important. If he knows this stuff, it is likely because
    is always pushing the envelope. This may or may not be the sort of guy
    you want. If you want him maintaining applications, you don't want an
    envelope-pusher guy. You want someone happy to write very mainstream
    pedestrian code.


    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Again taking new Java programming contracts.
    Roedy Green, Sep 30, 2005
    #19
  20. Roedy Green wrote:
    > On Thu, 29 Sep 2005 20:19:51 +0100, Thomas Hawtin
    > <> wrote or quoted :
    >
    >
    >>You will find it extremely difficult to find someone who
    >>knows the obscure but doesn't have a good general understand of
    >>programming and engineering principles.

    >
    >
    > Yes, but on the other hand, you may find that person a ditherer who
    > can't get on with real work. He is always fiddling with things that
    > are not that important. If he knows this stuff, it is likely because
    > is always pushing the envelope. This may or may not be the sort of guy
    > you want. If you want him maintaining applications, you don't want an
    > envelope-pusher guy. You want someone happy to write very mainstream
    > pedestrian code.


    That's a variation on the gold-plating myth. It's a popular one with
    managers who see a mock-up but don't understand that nothing works. I
    understand it doesn't really happen in real life. Unless you count
    witting tests and the such like.

    So, you'd go for pedestrian, copy & paste guy?

    Suppose you were developing a product and it was decided that testing
    should be written down as steps to follow in Word documents with visual
    checking of the results. Would you prefer the engineers going for that,
    or the one who spends an hour or two adapting the automated testing
    framework?

    The people you have to look out for is those who use inappropriate
    "technology" to put on their CV. Those people who just have to use OLE
    or EJB or whatever.

    Tom Hawtin
    --
    Unemployed English Java programmer
    http://jroller.com/page/tackline/
    Thomas Hawtin, Sep 30, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michelle
    Replies:
    0
    Views:
    389
    Michelle
    Sep 27, 2005
  2. CBFalconer
    Replies:
    17
    Views:
    793
    kyle york
    Nov 6, 2006
  3. Florian Thiel
    Replies:
    0
    Views:
    288
    Florian Thiel
    Jun 27, 2007
  4. Shaguf
    Replies:
    0
    Views:
    811
    Shaguf
    Nov 28, 2008
  5. Florian Thiel
    Replies:
    0
    Views:
    90
    Florian Thiel
    Jun 26, 2007
Loading...

Share This Page