The Closure Controversy

[Jon Harrop]

My day job is programming in modern statically-typed functional programming
languages like OCaml and F#. In my spare time in the evenings I like to
expand my mind by learning about really different languages like Java.

Tonight, I watched a fascinating lecture by Joshua Bloch called the "The
Closure Controversy":

http://www.parleys.com/display/PARLEYS/The+Closures+Controversy

From my point of view, this lecture is absolutely mind boggling and I even
found myself shouting at the screen a couple of times.

My final feeling about Joshua's statements is one of agreement. Trying to
retrofit real functional features onto Java is completely futile and cannot
yield anything usable. To make such things usable you must completely
overhaul the language, fixing the fundamental problems first and, if you do
that, why not doing it properly by simply building a whole new language for
the JVM?

I thought Joshua's welcome slides that described how Java was built to be
used by industry and not as a PhD thesis in academia was right on the
money. Consequently, I was very surprised to see Joshua recommend the Scala
programming language, as this is academic research.

This got me thinking, industry would benefit enormously from the creation of
a modern statically typed functional programming language that combined the
tried and tested benefits of languages like OCaml and Haskell with the JVM
platform. Will Sun or someone else build such a thing?

Microsoft are already doing so for .NET with F#, of course.

 

[SadRed]

My day job is programming in modern statically-typed functional programming
languages like OCaml and F#. In my spare time in the evenings I like to
expand my mind by learning about really different languages like Java.

Tonight, I watched a fascinating lecture by Joshua Bloch called the "The
Closure Controversy":

http://www.parleys.com/display/PARLEYS/The+Closures+Controversy

From my point of view, this lecture is absolutely mind boggling and I even
found myself shouting at the screen a couple of times.

My final feeling about Joshua's statements is one of agreement. Trying to
retrofit real functional features onto Java is completely futile and cannot
yield anything usable. To make such things usable you must completely
overhaul the language, fixing the fundamental problems first and, if you do
that, why not doing it properly by simply building a whole new language for
the JVM?

I thought Joshua's welcome slides that described how Java was built to be
used by industry and not as a PhD thesis in academia was right on the
money. Consequently, I was very surprised to see Joshua recommend the Scala
programming language, as this is academic research.

This got me thinking, industry would benefit enormously from the creation of
a modern statically typed functional programming language that combined the
tried and tested benefits of languages like OCaml and Haskell with the JVM
platform. Will Sun or someone else build such a thing?

Microsoft are already doing so for .NET with F#, of course.

fixing the fundamental problems first

What are the fundamental problems?

 

[Lew]

I thought Joshua's welcome slides that described how Java was built to be
used by industry and not as a PhD thesis in academia was right on the
money. Consequently, I was very surprised to see Joshua recommend the Scala
programming language, as this is academic research.

This got me thinking, industry would benefit enormously from the creation of
a modern statically typed functional programming language that combined the
tried and tested benefits of languages like OCaml and Haskell with the JVM
platform. Will Sun or someone else build such a thing?

Microsoft are already doing so for .NET with F#, of course.

Like MS, the new Java lives well with other languages. Pretty soon we'll all
be singing at La Scala, I wager.

Puns aside, given the ability to blend with other languages, and its use of
non-functional idioma like anonymous inner claases to replace "true closures"
(everybody pause in reverence), there's no real reason for Java itself to
change. Java doesn't need to be the ideal language - just blend it with a
JVM-ized Rhino or Scala or what-all. If the typical Java way doesn't suit,
you divide your app along domain boundaries that lend themselves to transition
into other languages.

Even that might not be necessary. The Java idioms actually do support
creation, on time, of software that will fulfill requirements with
reliability. While inner classes are a bit tricky and arguably ugly, they
work for pretty much anything you need, and frankly, are easier to read and
understand as a maintainer. Closure syntax is slick as all get-out, surely it
is, but it's got its own readability challenges.

The object oriented syntax of a Java anonymous inner class can be made to work
for you, both for the self-documentation nature of the source and to help the
Hotspot optimizer see opportunities. You see an override method, like
accept() for a FileFilter, that is going to iterate its operation over a whole
set of candidate File entries. Very closure-like. But instead of a mathlike
formula mapping a file to a boolean, you have a function-like accept() with
words describing the logical condition for the acceptance. Those words will
refer to a private method that progressively shows more detail, thus not
hiding the purpose in a complex algebraic expression.

All in a hairy anonymous class declaration. Except, not so hairy as all that
really, once one's actually trained oneself as a competent Java programmer.

So what if the Java inner-class idiom is but a clumsy, unsophisticated
workaround for the pristine elegance of true closures, and lacks in certain
arcane reaches - Java reaches far enough to get the job done.

 

[Jon Harrop]

Java reaches far enough to get the job done.

In some markets perhaps but I don't believe that would be true for our work.
Everything we sell carves out a niche and, consequently, has no competition
in the marketplace.

If I wanted to redevelop our software in Java we'd need 10 Java developers
for each OCaml/F# developer. Even if we could afford that outlay and risk,
I doubt Java versions would ship enough units to make any profit!

 

[Jon Harrop]

What are the fundamental problems?

The single biggest omission is type inference, the absence of which is
compounded by Java's unsuitable syntax for types.

Joshua gives examples of some trivial functions written in Java+closures
syntax:

static <A1, A2, R> { A1 => { A2 => R } } curry({A1, A2 => R} fn {
return { A1 a1 => A2 a2 => fn.invoke(a1, a2) } };
}

In OCaml this is just:

let curry f a1 a2 = f(a1, a2)

As Joshua says:

"Is this how you want Java programs to look?"

Due to the lack of type inference, the Java code is littered with lots of
types. In contrast, no types appear in the OCaml at all. As you can
imagine, trying to write anything non-trivial quickly becomes a practical
impossibility without type inference.

There are lots of other problems as well:

.. Java's checked exceptions do not scale in the presence of closures.
.. Syntactic burden for function-heavy code.
.. Arrays don't work: need to fix Java's type system.
.. Autoboxing doesn't work: the eight primitive types should not even exist.
....

There are also some more fundamental problems, like the lack of tail calls
in the JVM itself.

The only logical conclusion is that Java should be left alone and something
better should be implemented to replace it in the future. Scala is one
option but I'd rather see something tried and tested implemented on the JVM
rather than a completely new academic research language.

 

[Mark Thornton]

There are also some more fundamental problems, like the lack of tail calls
in the JVM itself.

That one might be addressed by the MultiLanguageVM project. I suspect it
could be added with rather less anguish and controversy than changes to
the language.

http://mail.openjdk.java.net/pipermail/announce/2007-October/000016.html

Mark Thornton

 

[Lew]

If I wanted to redevelop our software in Java we'd need 10 Java developers
for each OCaml/F# developer. Even if we could afford that outlay and risk,
I doubt Java versions would ship enough units to make any profit!

Do you have, oh, I don't know, any actual, replicable evidence for that ratio?

 

[Stefan Ram]

something better [than Java ]should be implemented
to replace it in the future.

Java now is the most successful language/platform
(6 million developers, 3.3 billion Java devices)
in the history of computing.

Language implementations with what you ask for (tail call
optimization, type inference, ...) already exist.
So everyone is already free to replace Java by such
an implementation right now.

Many Poeple just have chosen not to use such »better«
languages and to use Java instead.

So your problem might not be technical but social.
Technically, the »better language« already is there.
Socially, people choose not to use it often.

If you are sure you know a language that is so much
»better« than Java, and you use it for your own projects,
than you have a strong advantage over all the companies
using Java.

Wouldn't it be a good idea, then, just to use this advantage
to create great products with less effort? Since it gives
you such an edge, why do you want to help your competition
by convincing them to use it too? Actually, you should
keep it secret as long as possible.

If you really want to convince them, the outstanding success
of your projects with this »better language« should be more
convincing than words and praise could ever be.

 

[Hunter Gratzner]

Do you have, oh, I don't know, any actual, replicable evidence for that ratio?

No, he hasn't. Remember, Dr. Jon Harrop's first appearance in this
group was to disseminate propaganda for the language Scala and then
some spam for a book about Scale he happened to (co)author.

Dr "I have a PhD in computational physics from the university of
Cambridge" Harrop is only here in this group to demonstrate his
alleged superiority and to show us stupid Java programmers how
superior beings would do things. I consider him to be in the same
group as Xah Lee and our broken record "implied insult deleted"
groupie.

There is much to be said about closures in Java, but I don't consider
Dr Harrop's biased comments a valuable contribution to such
discussions.

 

[Joe Attardi]

I consider him to be in the same group as Xah Lee and
our broken record "implied insult deleted" groupie.

None of the nasty things you have implied about Paul Derbyshire are in
any way true.



Kidding, of course...!

 

[Jeff Higgins]

Amen. Well said. Thank you.

 

[Wildemar Wildenburger]

None of the nasty things you have implied about Paul Derbyshire are in
any way true.



Kidding, of course...!

Not that i have any form of authority here, but I would like to see the
Twisted-bashing exclusively contained in the SWT-thread. Just so the
fire won't spread too much.

humbly,
/W

 

[Jon Harrop]

Do you have, oh, I don't know, any actual, replicable evidence for that
ratio?

Yes. That is the number of Java programmers who get replaced by each
OCaml/F# developer whenever a company takes the plunge to use these
languages.

Look at XenSource, for example. 30 Java programmers in the USA were replaced
by 3 OCaml programmers in the UK. The Java programmers failed to ship
anything at all in 3 years and the OCaml programmers shipped something of
such high quality in only 6 months that the company then sold for $500M.

Lots of companies have done similar things. We're one of them, reborn from
C++ development a decade ago.

This is nothing new. Java offers substantial productivity gains over C.

 

[Roedy Green]

Java now is the most successful language/platform
(6 million developers, 3.3 billion Java devices)
in the history of computing.

It not so much Java the language that is so popular, it is Java the
class library suite and Java the JVM that runs on many platforms
without modification.

It should be relatively easy to redo the Java language, keeping
everything else the same.

At some point surely Sun will take a second run at generics, without
type erasure. Just how does Sun then deprecate the first crack?


It might be a good time to introduce Java II that has many syntax
tidies and some incompatibilities.

 

[Jon Harrop]

It not so much Java the language that is so popular, it is Java the
class library suite and Java the JVM that runs on many platforms
without modification.
Exactly.

It should be relatively easy to redo the Java language, keeping
everything else the same.

Hopefully.

 

[Mark Thornton]

Look at XenSource, for example. 30 Java programmers in the USA were replaced
by 3 OCaml programmers in the UK. The Java programmers failed to ship
anything at all in 3 years and the OCaml programmers shipped something of
such high quality in only 6 months that the company then sold for $500M.

It is most likely that this says more about the programmers than it does
about the languages concerned. As it seems common to find that 5% of
programmers are more productive than the other 95%, if you can find
those 5% you can 'prove' any reasonable language is the best. Java's
success has brought with it a large coterie of uninspiring Java programmers.

Lots of companies have done similar things. We're one of them, reborn from
C++ development a decade ago.

Perhaps your enthusiasm is as important as the chosen language.

This is nothing new. Java offers substantial productivity gains over C.

That giant 'core' library is as important as the language itself. This
remains an advantage over many would be successors (other than JVM
hosted languages). .NET is even larger, but personally I would be
unhappy using it on anything but Windows.

Mark Thornton

 

[Jon Harrop]

something better [than Java ]should be implemented
to replace it in the future.

Java now is the most successful language/platform
(6 million developers, 3.3 billion Java devices)
in the history of computing.

In terms of market share, that could be a lot better.

Language implementations with what you ask for (tail call
optimization, type inference, ...) already exist.

Not on the JVM.

So everyone is already free to replace Java by such
an implementation right now.

No. Until better languages are implemented interoperably with Java on the
JVM, replacing Java with something else is not "free" but prohibitively
expensive.

Many Poeple just have chosen not to use such »better«
languages and to use Java instead.

Actually, most programmers don't use Java.

So your problem might not be technical but social.
Technically, the »better language« already is there.
Socially, people choose not to use it often.

If the problem were social the Java community would not be working so hard
to adopt features like closures.

If you are sure you know a language that is so much
»better« than Java, and you use it for your own projects,
then you have a strong advantage over all the companies
using Java.

You are assuming that we compete with companies that use Java but we do not.

Wouldn't it be a good idea, then, just to use this advantage
to create great products with less effort?

Yes. We have used a variety of different languages (including Java) to
create products. However, many of our products cannot be written by hand in
a language like Java because it requires too much work to write and
maintain.

Since it gives
you such an edge, why do you want to help your competition
by convincing them to use it too?

We don't have any competition so that doesn't concern us.

Actually, you should keep it secret as long as possible.

I would rather build a better future for everyone: more choice of languages
and more JVM users.

If you really want to convince them, the outstanding success
of your projects with this »better language« should be more
convincing than words and praise could ever be.

That already happened.

 

[Jon Harrop]

It is most likely that this says more about the programmers than it does
about the languages concerned. As it seems common to find that 5% of
programmers are more productive than the other 95%, if you can find
those 5% you can 'prove' any reasonable language is the best. Java's
success has brought with it a large coterie of uninspiring Java
programmers.

Exactly. Java is for the 95% but there's no reason the JVM couldn't be
reused to help the 5% do the other half of the work.

That giant 'core' library is as important as the language itself. This
remains an advantage over many would be successors (other than JVM
hosted languages). .NET is even larger, but personally I would be
unhappy using it on anything but Windows.

Yes. We use F# extensively and I would like to see exactly that kind of
innovation on the JVM. The requirements seem quite mundane to me: the
research was done and dusted a long time ago...

 

[Stefan Ram]

If the problem were social the Java community would not be
working so hard to adopt features like closures.

Remarkably, Guy Steele claimed that once the Java community
was working hard to /remove/ features like closures.

»Guy Steele wrote:

Actually, the prototype implementation *did* allow
non-final variables to be referenced from within
inner classes. There was an outcry from *users*,
complaining that they did not want this!«

http://madbean.com/2003/mb2003-49/

 

[Lew]

Remarkably, Guy Steele claimed that once the Java community
was working hard to /remove/ features like closures.

»Guy Steele wrote:

Actually, the prototype implementation *did* allow
non-final variables to be referenced from within
inner classes. There was an outcry from *users*,
complaining that they did not want this!«

http://madbean.com/2003/mb2003-49/

Who is this "Java community" really? Do those who claim to speak for it
really do so?

There wouldn't be a "Closure Controversy", as this thread is entitled, if
support for it were so universal.

This Java programmer happens to like anonymous class syntax. While I have
seen what closures can do for those who initially *write* code, it seems less
kind to maintainers. Perhaps from being clean-up for so many code bases over
the years, I have grown leery of too-clever, too-compact idioms that require a
maintainer to hunt all over source just to see what "x,y => majorOp( x
).convolve( y )" means. At the expense of initial verbosity on the part of
the programmer, who otherwise might be sorely tempted to leave out critical
self-documentation, Java maintainers get a literate, connected and somewhat
self-explanatory inner class.

Those who complain that the syntax is "ugly" are probably right, but it's hard
to put an engineering value on such an assessment. Besides, properly indented
inner classes aren't so hard to read, and in the end are easy enough to
maintain for someone schooled in the idiom.

Arguably, closure syntax is just as transparent given similar familiarity.
The question is whether their introduction into Java provides enough benefit
to offset feature creep (leaving aside Java II, just when you thought it was
safe to dip into the JVM again), or whether current idioms do enough that we
will just live without closures. It's not so much if they're better, as
whether they're better enough.

Perhaps those intimately familiar with the reasons why Guy Steele's "*users*"
raised such an outcry would share them with us.