First class developer: who ?

[Arved Sandstrom]

Threads can somewhat be avoided in desktop apps, but threads
are usually unavoidable in server apps.

Arne

And it's simply amazing how many of them work and how few bugs have
anything to do with concurrency even though *everything* is running in a
multi-threaded environment.

At one client site where I have been for about a year and a half the
developers, internal and external, have probably fixed over a thousand
defects in that time. In two fairly good-sized J2EE apps. Not one - I
repeat, not one - defect has had anything to do with concurrency issues.
Good thing I didn't read that Sun blurb - I might have avoided working
with J2EE at all.

For that same client I wrote a moderately complex inherently
multi-threaded socket server that went into production over a year ago.
It has had no unscheduled downtime, and has had less than five defects
logged against it, none of them serious, and none of them having
anything to do with concurrency problems. Good thing I didn't read that
Sun blurb first - I would have avoided java.util.concurrent classes like
the plague, because none of them could possibly have worked well.

While working for an employer back about 7 or 8 years ago I was in
charge of the in-house servlet engine, which was partially written in C
but mostly in Java. Plenty of multi-threading happening there. Given how
incredibly difficult writing this kind of code is, according to Sun, I'm
amazed at how few concurrency problems we had back then. We must have
just lucked out, I guess.

The stated intent of JCIP, and other books like it, is to arm developers
with the information they need to write reliable and maintainable
concurrent apps. Clearly a fair few people are able to do precisely
that. So are the authors of JCIP, and all those programmers out there
who have actually demonstrably written reliable and maintainable
concurrent applications, all full of it? I don't think so.

AHS

 

[Mike Schilling]

For that same client I wrote a moderately complex inherently
multi-threaded socket server that went into production over a year
ago. It has had no unscheduled downtime, and has had less than five
defects logged against it, none of them serious, and none of them
having anything to do with concurrency problems. Good thing I didn't
read that Sun blurb first - I would have avoided java.util.concurrent
classes like the plague, because none of them could possibly have
worked well.

That is, you're using classes designed, created, implemented and debugged by
Sun that have been exercised so heavily that the commonly encountered bugs
have ben found If you had to implement something like a
ConcurrentLinkedQueue from scratch, do you think it would just work, or
might there be subtle, difficult-to-detect bugs in it?

 

[Arved Sandstrom]

That is, you're using classes designed, created, implemented and debugged by
Sun that have been exercised so heavily that the commonly encountered bugs
have ben found If you had to implement something like a
ConcurrentLinkedQueue from scratch, do you think it would just work, or
might there be subtle, difficult-to-detect bugs in it?

In this _particular_ case I chose classes designed, created, implemented
and debugged by Sun, yes. Normally, now, I'd be inclined to make that
same choice rather than create my own low-level stuff from scratch.

As to the question of whether my own implementations of classes that are
in java.util.concurrent would have subtle, difficult-to-detect bugs in
them, I can't answer that in a way that would probably satisfy you. I
could make the point that prior to using java.util.concurrent classes
that, in one of the other projects I mentioned, that we were using
strictly the lower-level Java concurrency facilities (the Object and
Thread methods), and although there were bugs related to our concurrency
implementation, the number and difficulty of them wasn't of such a
nature as to make them stand out compared to defects encountered in
other areas of the code. In fact it proved not to be all that tough to
write a reliable server using the lower-level Java facilities.

The more complicated logic there, through the life of the HTTP/S
server/servlet engine, had to do with properly implementing HTTP and the
servlet API. Not the concurrency.

But we did have good programmers. That surely makes a difference. I've
already stated that I wouldn't expect junior programmers to be writing
good concurrency code. I wouldn't expect them to be writing particularly
good code in any problem area, quite frankly.

I might also add that in this same socket server that you called out,
the one where I chose to use java.util.concurrent classes for the
concurrency problem, the majority of the code has nothing whatsoever to
do with concurrency. A majority of the most complicated code has to do
with classloader management so as to allow hot-unloading/reloading of
classes without stopping/starting/restarting the server. I wasn't
familiar with OSGi at the time so I went that route. Point being, _that_
code is functioning very reliably as well, over 15 months now in
high-volume 24/7 use, so maybe just maybe if I'd written that
concurrency code using lower-level Java facilities I might have known
what I was doing - since I've spent a lot of time doing professional
development on the subject - and there wouldn't have been a whole bunch
of subtle and difficult-to-detect concurrency defects.

Perhaps I can rephrase my main argument. I'm of the belief that
concurrency problems, when we consider their difficulty, fall into a
spectrum just like problems in every other area of programming do -
easy, moderately difficult, and difficult. I also believe that when
concurrency problems are tackled by intermediate/senior developers who
have a decent knowledge of concurrency programming that most of those
problems are easy or moderately difficult...just like most of the
non-concurrency problems.

Let me ask you this, Mike. I would have to assume that you've done
substantially the same reading I have (nature of, if not the same
titles), and likely also have considerable experience in concurrent
programming. Let's take Java concurrent programming as an example,
because of the NG we're in, and examine the JCIP book specifically.
After reading and re-reading that book, is there anything in it that
leads you to believe that only a tiny elite of superstar developers can
ever "get" concurrent programming in Java?

AHS

 

[Lew]

Let me ask you this, Mike. I would have to assume that you've done
substantially the same reading I have (nature of, if not the same
titles), and likely also have considerable experience in concurrent
programming. Let's take Java concurrent programming as an example,
because of the NG we're in, and examine the JCIP book specifically.
After reading and re-reading that book, is there anything in it that
leads you to believe that only a tiny elite of superstar developers can
ever "get" concurrent programming in Java?

This is an irresolvable discussion. Many programmers find concurrency
programming to be subtle and difficult compared to other areas, Arved's
experience notwithstanding. Arved's facility in that area will never convince
them otherwise, just as their difficulties will never make concurrency seem
especially difficult to Arved. There are a number of programmers out there
who have trouble getting a (single-threaded) 'for' loop correct. There's no
objective way to say /a priori/ that such-and-such area of programming is
especially difficult because it's so dependent on an individual's talents.

That said, across the industry there are many programmers who find concurrency
difficult and subtle. Perhaps their talents are that different from Arved's,
or perhaps like the hapless 'for'-loop schmo they just haven't grokked it yet
and they'll improve when they do. It is objective that concurrency bugs are
harder to diagnose, test and debug than many other kinds, and that elevates
the importance of catching them at the source rather than at runtime. It
behooves us to form mental models of concurrency that let us program for it as
effectively and effortlessly as Arved, Josh Bloch, Doug Lea and Brian Goetz do.

 

[Mike Schilling]

Let me ask you this, Mike. I would have to assume that you've done
substantially the same reading I have (nature of, if not the same
titles), and likely also have considerable experience in concurrent
programming. Let's take Java concurrent programming as an example,
because of the NG we're in, and examine the JCIP book specifically.
After reading and re-reading that book, is there anything in it that
leads you to believe that only a tiny elite of superstar developers
can ever "get" concurrent programming in Java?

I've never said that. Nor has Patricia, nor anpne else in this thread.
What we have said is that concurrency introduces compleixty and bugs that
are difficult to reproduce, so that, compared to single-threaded programs,
more effort is involved in producing a product of the same quality. So I'll
ask you again: do you think you could code ConcurrentLinkedQueue from
scratch and be satisfied that it's bug-free in roughly the same amount of
time as, say, TreeMap?

 

[Arved Sandstrom]

This is an irresolvable discussion. Many programmers find concurrency
programming to be subtle and difficult compared to other areas, Arved's
experience notwithstanding. Arved's facility in that area will never
convince them otherwise, just as their difficulties will never make
concurrency seem especially difficult to Arved. There are a number of
programmers out there who have trouble getting a (single-threaded) 'for'
loop correct. There's no objective way to say /a priori/ that
such-and-such area of programming is especially difficult because it's
so dependent on an individual's talents.

That said, across the industry there are many programmers who find
concurrency difficult and subtle. Perhaps their talents are that
different from Arved's, or perhaps like the hapless 'for'-loop schmo
they just haven't grokked it yet and they'll improve when they do. It
is objective that concurrency bugs are harder to diagnose, test and
debug than many other kinds, and that elevates the importance of
catching them at the source rather than at runtime. It behooves us to
form mental models of concurrency that let us program for it as
effectively and effortlessly as Arved, Josh Bloch, Doug Lea and Brian
Goetz do.

Well, here's the thing - it's not just me. Clearly fairly large numbers
of developers are getting it, because there's a lot of pretty reliable
concurrent code out there. And it's not like Josh Bloch, Doug Lea and
Brian Goetz (and Joe Duffy going apeshit on the .NET side of things)
wrote all of it.

If I ran across a reasonably senior developer who, when presented with a
concurrent programming problem, said that he or she was finding
multi-threading complicated and hard to reason about, I'd simply ask
them how much experience they have with concurrent programming. And how
much study they have devoted to it. If the answer is "none" or 'not
much", well, duh. I wouldn't expect any developer to be particularly
good at anything if they have little knowledge of it.

I will absolutely agree on one thing, and that's that in concurrent
programming, more so than in most other types of programming, you'd
better have your act together when it comes to being a rounded software
developer. In many other areas you can be sloppy and write sloppy
ill-designed code, and user testing finds your defects, and you fix
them...not so in concurrent programming.

But the fact that the nature of programmer-created defects in concurrent
code - let me re-emphasize that, _programmer-created defects_ in
concurrent code - is such that they are harder to reason about than
defects in most non-concurrent code does not equate to an observation
that concurrent programming is actually that much more difficult.

We should maybe separate this discussion into two parts: are people
saying that correct concurrent programming is very difficult, or are
they saying that debugging badly written concurrent code is very
difficult? I certainly won't argue with the latter statement.

AHS

 

[Arved Sandstrom]

I've never said that. Nor has Patricia, nor anpne else in this thread.
What we have said is that concurrency introduces compleixty and bugs that
are difficult to reproduce, so that, compared to single-threaded programs,
more effort is involved in producing a product of the same quality.

Perhaps I'm reading more into what you guys are saying than I should,
but the sense I am getting from your statements is that concurrency
introduces _much_ more complexity and difficulty, so much more that
simple problems become difficult ones, and difficult single-threaded
problems become almost infernally complicated with concurrency.

My argument is more along the lines that introducing concurrency into a
problem may add minor, moderate or major levels of extra difficulty, but
that I see no evidence that it's always or even often got to be the latter.

So I'll
ask you again: do you think you could code ConcurrentLinkedQueue from
scratch and be satisfied that it's bug-free in roughly the same amount of
time as, say, TreeMap?

Looking at the Java 6 versions of the APIs for both, and given that you
imposed no other requirements other than that both be bug free, I'd say yes.

After all, given no requirements other than correctness, I don't have to
produce a ConcurrentLinkedQueue that is as performant or efficient or
elegant, with extra classes like
java.util.concurrent.atomic.AtomicReferenceFieldUpdater and
sun.misc.Unsafe, as what Doug Lea wrote.

AHS

 

[Arved Sandstrom]

Arved Sandstrom wrote:
[ SNIP ]

After all, given no requirements other than correctness, I don't have to
produce a ConcurrentLinkedQueue that is as performant or efficient or
elegant, with extra classes like
java.util.concurrent.atomic.AtomicReferenceFieldUpdater and
sun.misc.Unsafe, as what Doug Lea wrote.

I meant to add, there isn't a single use of the "synchronized" keyword
in ConcurrentLinkedQueue. I consider myself a competent concurrency
programmer, but not a god like Doug Lea or Brian Goetz. My estimate was
based on being allowed to use "synchronized", which as a first cut is a
sensible thing to do.

AHS

 

[Mike Schilling]

Arved Sandstrom wrote:
[ SNIP ]

After all, given no requirements other than correctness, I don't
have to produce a ConcurrentLinkedQueue that is as performant or
efficient or elegant, with extra classes like
java.util.concurrent.atomic.AtomicReferenceFieldUpdater and
sun.misc.Unsafe, as what Doug Lea wrote.

I meant to add, there isn't a single use of the "synchronized" keyword
in ConcurrentLinkedQueue. I consider myself a competent concurrency
programmer, but not a god like Doug Lea or Brian Goetz. My estimate
was based on being allowed to use "synchronized", which as a first
cut is a sensible thing to do.

No, that's cheating The whole point of ConcurrentLinkedQueue is to
achieve concurrency without locking the queue down. And you know what?
It's complicated.

 

[Arved Sandstrom]

Arved Sandstrom wrote:
[ SNIP ]

After all, given no requirements other than correctness, I don't
have to produce a ConcurrentLinkedQueue that is as performant or
efficient or elegant, with extra classes like
java.util.concurrent.atomic.AtomicReferenceFieldUpdater and
sun.misc.Unsafe, as what Doug Lea wrote.

I meant to add, there isn't a single use of the "synchronized" keyword
in ConcurrentLinkedQueue. I consider myself a competent concurrency
programmer, but not a god like Doug Lea or Brian Goetz. My estimate
was based on being allowed to use "synchronized", which as a first
cut is a sensible thing to do.

No, that's cheating The whole point of ConcurrentLinkedQueue is to
achieve concurrency without locking the queue down. And you know what?
It's complicated.

I don't consider this cheating at all. Your language was, "If you had to
implement something like a ConcurrentLinkedQueue from scratch". You'll
note the "something like a".

How "like"? If I'm permitted to go off the published API (e.g.
http://java.sun.com/javase/6/docs/api/java/util/concurrent/ConcurrentLinkedQueue.html)
, which certainly meets your stated requirements (in fact it's even more
strict than what you asked for), then using synchronized methods is
perfectly adequate. From a software engineering standpoint I'd be silly
to produce a Lamborghini when you only asked for a Honda.

If you wish to impose additional requirements, namely on the
implementation, feel free. I'll point out, though, that there's still a
lot of room between a straight synchronize-all-methods approach (OK,
it's not quite that simple, but it's close) and the Ultimate Engineering
that we see in the actual JDK class. My argument is that substantial
improvements, over and above the simplest synchronize-all-methods
approach, can be achieved with approaches considerably less refined and
intricate than the Lea implementation.

You can guess that the simplest approach, one based on synchronizing the
top-level methods, doesn't take long at all. Just for giggles I wrote a
new ConcurrentLinkedQueue from scratch yesterday, complete with about 25
JUnit tests so far, and one load test that runs through a reasonably
complicated queue operations sequence for many threads, and it's doing
just fine. No doubt in my mind that you and many other people would have
knocked out the same thing in about the same amount of time.

I'll probably try some improvements in the direction that you suggest,
just for fun. As to whether an improved implementation would require
more time or about the same amount of time as TreeMap, that depends
entirely on what the performance requirements are. I think a good
concurrency programmer could implement adequate next-stage mechanisms
and not spend appreciably more time on this class than they would on
TreeMap. You should remember, the core logic of ConcurrentLinkedQueue is
not exactly all that complicated.

AHS