Taking advantage of clusters or multi CPUs?

[Kevin]

Hello,

I am just wondering that how can we make sure a java program can take
advantage of those cluster machines, or those with multi CPUs? ------ I
mean, besides write our code in a "proper" way, do we need to
"explicitly" add something into the code or tell something to the
compiler or JVM?

Any guide line for coding with multi CPUs or cluster machines?

Thanks!

 

[Knute Johnson]

Hello,

I am just wondering that how can we make sure a java program can take
advantage of those cluster machines, or those with multi CPUs? ------ I
mean, besides write our code in a "proper" way, do we need to
"explicitly" add something into the code or tell something to the
compiler or JVM?

Any guide line for coding with multi CPUs or cluster machines?

Thanks!

I asked this question a few weeks ago. I'm currently working on a big
project that has several large multi-threaded apps. We were concerned
about getting enough computer to make the project lively enough. We
ended up buying single board computers from Crystal Group. They have
two Intel 2.8Ghz Xeon dual core processors. They are really fast. But
back to the multi-threaded issue. I asked about that and the tech
advisers at Crystal said that the OS (in this case XP Pro) should take
care of the multi-threading issues. When we run our program and look at
the task manager, it shows four processors. The processor load is
fairly well balanced between all four. Sometime if we don't load it up,
it will show two of the processors with load and the other two not using
much. My programs are written using normal multi-threading code. I
didn't have to change anything that I wrote.

Hope that helps some .

 

[Kevin]

Thanks for that. I think that is good as far as to multi threading.

How about single threading? For example, if my program has a thread
that do a lot of computations, and these computations do not affect
each other. Say, the thread is scanning through a list of records, and
do some computation within each record.

One way is of course to use many threads to do this. But if I just put
them in one loop in one thread, will JVM and its compiler be "smart"
enough to realize it and use multi CPUs to do them in parallel?

The reason I am asking is that, as I remembered years back on my
compiler course in school, they said good compilers can know certain
codes can be paralleled, and the complier will tune the code
automatically. Is that available in today's common compliers (and
languages) such as in java?

Hum, my question may be some what off the topic for a Java group
though.

 

[Chris Uppal]

The reason I am asking is that, as I remembered years back on my
compiler course in school, they said good compilers can know certain
codes can be paralleled, and the complier will tune the code
automatically. Is that available in today's common compliers (and
languages) such as in java?

Not Java.

There are some special-purpose compilers for languages like Fortran than /can/
do that kind of thing, but they still need plenty of help from the programmer
to do it /well/.

There are languages designed from the ground up to parallelise too.

-- chris

 

[Roedy Green]

I am just wondering that how can we make sure a java program can take
advantage of those cluster machines, or those with multi CPUs? ------ I
mean, besides write our code in a "proper" way, do we need to
"explicitly" add something into the code or tell something to the
compiler or JVM?

There are two kinds of parallelism, pipelines when one CPU processing
an instruction stream slightly out of order and in parallel, but
logically it all works as if it had done it in order. You don't have
to do anything but buy an expensive CPU to get this benefit.

The other kind is like the Sun Niagara servers where they have 8
logically separate CPUs on single chip. The share some logic, but from
your point of view they process 8 instructions streams fully
simultaneously.

This also applies to hyperthreading chips that simulate two CPUS with
a great deal of shared logic.

For these to buy you anything you must either:
1. have other tasks running the background outside your JVM. They may
be non-java or other Java programs. Obviously you need lots of RAM for
hem.

2. you use threads and those threads done just lie around waiting for
each other. They are busy computing or doing i/o most of the time.

Imagine for example a HTML link checker. A multithreaded one could be
checking 30 links at a time. With a single thread it could only check
one at a time. See http://mindprod.com/projects/htmlbrokenlink.html
If you had 5 cpus, each could handle 6 threads on average.

Threads are not dedicated to a CPU. A CPU, when it becomes free, just
takes the next thread ready to continue executing.

 

[Roedy Green]

One way is of course to use many threads to do this. But if I just put
them in one loop in one thread, will JVM and its compiler be "smart"
enough to realize it and use multi CPUs to do them in parallel?

no. There is a fair overhead to creating a thread and starting it up.
You can't do it just for tiny bits of code. Further as soon as you
introduce threads you introduce terrible uncertainty as they meddle
with each other's variables. It requires a much higher level of skill.
It is still far from being automatic.

You want your threads to have ZERO interaction ideally.

See http://mindprod.com/jgloss/queue.html

for stereotyped ways of interacting that have been coded by the gods.

Rolling you own interactions is very difficult to get 100%. Getting it
90% is easy.

 

[Nigel Wade]

Hello,

I am just wondering that how can we make sure a java program can take
advantage of those cluster machines, or those with multi CPUs? ------ I
mean, besides write our code in a "proper" way, do we need to
"explicitly" add something into the code or tell something to the
compiler or JVM?

Any guide line for coding with multi CPUs or cluster machines?

Thanks!

Java will use multiple processors in SMP systems, as permitted by the underlying
OS. It can't use other nodes in a cluster which typically requires special
software which isn't built into Java.

 

[Nigel Wade]

Thanks for that. I think that is good as far as to multi threading.

How about single threading? For example, if my program has a thread
that do a lot of computations, and these computations do not affect
each other. Say, the thread is scanning through a list of records, and
do some computation within each record.

One way is of course to use many threads to do this. But if I just put
them in one loop in one thread, will JVM and its compiler be "smart"
enough to realize it and use multi CPUs to do them in parallel?
No.


The reason I am asking is that, as I remembered years back on my
compiler course in school, they said good compilers can know certain
codes can be paralleled, and the complier will tune the code
automatically. Is that available in today's common compliers (and
languages) such as in java?

Normally, only in very expensive parallelising compliers. Even then there are
only some very specific types of code which can be automatically parallelised,
it normally requires specific parallel constructs to be used in the code (my
only knowledge of parallelising compilers is for FORTRAN, it may be different
for other languages). It's also quite easy to write code which takes longer to
execute on multiple processors than it does on one...

 

[Oliver Wong]

How about single threading? For example, if my program has a thread
that do a lot of computations, and these computations do not affect
each other. Say, the thread is scanning through a list of records, and
do some computation within each record.

One way is of course to use many threads to do this. But if I just put
them in one loop in one thread, will JVM and its compiler be "smart"
enough to realize it and use multi CPUs to do them in parallel?

The reason I am asking is that, as I remembered years back on my
compiler course in school, they said good compilers can know certain
codes can be paralleled, and the complier will tune the code
automatically. Is that available in today's common compliers (and
languages) such as in java?

Hum, my question may be some what off the topic for a Java group
though.

A lot of people have been saying "No, there's no such thing." I also
acknowledge that what you're asking for is difficult enough that I would be
"surprised" to discover a compiler which can automatically insert threading
when appropriate, but I don't presume to know of all the latest and greatest
in compiler advancements.

You might want to redirect your question to comp.compilers.

- Oliver

 

[Dimitri Maziuk]

Kevin sez:

Thanks for that. I think that is good as far as to multi threading.

How about single threading? For example, if my program has a thread
that do a lot of computations, and these computations do not affect
each other. Say, the thread is scanning through a list of records, and
do some computation within each record.

There 2 kinds of parallel: single-instruction-multiple-data and
multiple-instructions-multiple-data, they require different kinds
of hardware. What you're talking about is SIMD (same computation
on different records) and unless you happen to have a MasPar in
your basement, you're SOL. These things come with specialized
languages (or at least specialized C dialects) and specialized
programming techiniques (e.g. "wavefront").

Your regular SMP machines and multi-core CPUs are MIMD, designed
for multiple threads/processes, not for SIMD tasks. You could
potentially do SIMD on them if you could bypass everything
including on-chip instruction pipeline & parallelizer and get
direct access to chip's cores. Definitely not in Java, for
obvious reasons.

One way is of course to use many threads to do this. But if I just put
them in one loop in one thread, will JVM and its compiler be "smart"
enough to realize it and use multi CPUs to do them in parallel?
Nope.

The reason I am asking is that, as I remembered years back on my
compiler course in school, they said good compilers can know certain
codes can be paralleled, and the complier will tune the code
automatically. Is that available in today's common compliers (and
languages) such as in java?

IIRC Ada has "parbegin ... parend" blocks, allowing the programmer
to mark code segments for parallel execution. They may become common
if IBM's new playstation chip takes off (PPC "dispatcher" core + a
bunch of SIMD "processing" cores). Then come parallelizing compilers...

Dima

 

[Roedy Green]

if you could bypass everything
including on-chip instruction pipeline & parallelizer and get
direct access to chip's cores. Definitely not in Java, for
obvious reasons.

most chips nowadays come with one or more specialised auxiliary
processors, DSPs which are a sort of miniature array processor. These
are primarily intended for video or audio processing. You might see
Java using them, but only a in a native method. The most common place
you see them used are in CODECs.

see http://mindprod.com/jgloss/dsp.html