java based supercomputer

[sav]

I am trying an ambitous project of trying to adapt a piece of code
that i came accross on an IBM website. It uses psuedo remote threads
and is a way of utilising RMI, multithreading and will develop a java
based supercomuting technique. I wish to use a similar method for
checking the correlation beteween an array of data and another array
comprised of combining 8 other arrays of data, so by using a
straightforward for next loop will take forever but by utilising this
java psuedo remote threads will take a considerrably less time.
Howeve, i am relatively new to java but i learn fast. I need someont
to help me achieve this, i am willing to pay. I hacve the code
available for review

 

[Daniel Pitts]

I am trying an ambitous project of trying to adapt a piece of code
that i came accross on an IBM website. It uses psuedo remote threads
and is a way of utilising RMI, multithreading and will develop a java
based supercomuting technique. I wish to use a similar method for
checking the correlation beteween an array of data and another array
comprised of combining 8 other arrays of data, so by using a
straightforward for next loop will take forever but by utilising this
java psuedo remote threads will take a considerrably less time.
Howeve, i am relatively new to java but i learn fast. I need someont
to help me achieve this, i am willing to pay. I hacve the code
available for review

This first question, which isn't even Java specific, is:
Does your algorithm lend itself well to paralellization? You could
find out by running it with normal Thread's on a multiprocessor
system. If this improves performance, then you can proceed to the next
questions.

The second question, still not Java specific:
When you split the computation between N threads, how much
communication is there between those threads? If the communication
between threads has higher latency or limited bandwidth, how does that
effect your algorithms throughput?

Java specific question: Have you profiled your existing "slow" program
to see what is taking the most time? It might not be what you
expect.

Also, have you considered finding a less naive algorithm? Without
more details about your goal I can only speculate, but it might be
that your approach is an O(N^3) algorithm, when in fact there may
exist an O(n*logn) algorithm.

So, before trying to cross machine boundaries, or even thread
boundaries, it might be worth looking into other algorithms that speed
up your process.

 

[sav]

This first question, which isn't even Java specific, is:
Does your algorithm lend itself well to paralellization? You could
find out by running it with normal Thread's on a multiprocessor
system. If this improves performance, then you can proceed to the next
questions.

The second question, still not Java specific:
When you split the computation between N threads, how much
communication is there between those threads? If the communication
between threads has higher latency or limited bandwidth, how does that
effect your algorithms throughput?

Java specific question: Have you profiled your existing "slow" program
to see what is taking the most time? It might not be what you
expect.

Also, have you considered finding a less naive algorithm? Without
more details about your goal I can only speculate, but it might be
that your approach is an O(N^3) algorithm, when in fact there may
exist an O(n*logn) algorithm.

So, before trying to cross machine boundaries, or even thread
boundaries, it might be worth looking into other algorithms that speed
up your process.

Hi i did send you a full email to your email address did you get it?

 

[Daniel Pitts]

Hi i did send you a full email to your email address did you get it?

No. Three reasons. The first is that my home computer has been
disabled for a few weeks.
The second reason is that the given e-mail address gets so much spam
that I usually just delete it with out looking.
The third reason is that I didn't invite you to e-mail me, please keep
the discussion here unless invited to otherwise.

As far as I know, no one here gets paid to answer questions on this
newsgroup. On the other hand, many of us DO get paid to answer
questions one-on-one. If you want free advice, post here; if you want
to pay for solutions, find a contractor.

 

[sav]

No. Three reasons. The first is that my home computer has been
disabled for a few weeks.
The second reason is that the given e-mail address gets so much spam
that I usually just delete it with out looking.
The third reason is that I didn't invite you to e-mail me, please keep
the discussion here unless invited to otherwise.

As far as I know, no one here gets paid to answer questions on this
newsgroup. On the other hand, many of us DO get paid to answer
questions one-on-one. If you want free advice, post here; if you want
to pay for solutions, find a contractor.

ok cool, weel first to answer your questions, i am pretty sure that
the algorithm will be quite easy to parralelise if that is the correct
term, the only bottleneck i can see is checking the correlation value
against the highest rank, ie the algorithm basically is a for next
loop BUT with billions of calcs to perform, the bottlenck will
obviously be when the calcs need to check for a higher correlation
against the current highest rank, however with the java program i have
seen it seesm to perform many of the loop functions simoultaneously,
ie i have 8 arrays of of which perform 10^3 calculations so many of
these calcs can be carried out concurrently.

The current system runs in VBA not in java so i havent run in java
yet.

Look forward to your reply

 

[Daniel Pitts]

ok cool, weel first to answer your questions, i am pretty sure that
the algorithm will be quite easy to parralelise if that is the correct
term, the only bottleneck i can see is checking the correlation value
against the highest rank, ie the algorithm basically is a for next
loop BUT with billions of calcs to perform, the bottlenck will
obviously be when the calcs need to check for a higher correlation
against the current highest rank, however with the java program i have
seen it seesm to perform many of the loop functions simoultaneously,
ie i have 8 arrays of of which perform 10^3 calculations so many of
these calcs can be carried out concurrently.

The current system runs in VBA not in java so i havent run in java
yet.

Look forward to your reply

Okay, so if I understand correctly:

You have N number of arrays to test;
Testing array A doesn't interact with the testing of array B.
Testing any array results in an easy-to-compare score.
So, you could create up to N threads, each calculates and records the
score.
All of the scores can then be easily compared to one another.

Since there are only 8 arrays (hence 8 scores), the sorting isn't too
hard to do.

How do you score array A? What do you mean by correlation? longest-
common-subsequence? number of similar/same elements? Something else?

 

[sav]

Okay, so if I understand correctly:

You have N number of arrays to test;
Testing array A doesn't interact with the testing of array B.
Testing any array results in an easy-to-compare score.
So, you could create up to N threads, each calculates and records the
score.
All of the scores can then be easily compared to one another.

Since there are only 8 arrays (hence 8 scores), the sorting isn't too
hard to do.

How do you score array A? What do you mean by correlation? longest-
common-subsequence? number of similar/same elements? Something else?

Nok you are quite close its a little more compliacted than that but
you have the basic idea, Array A values dont change they are fixed
upon loaded from en excel file, i have written the code for that,
array b is made up of values based upon the values from 8 other arrays
each array is a calculation of (10^3)*N calcs, ie array 1 does a clac
and is then added to the values of the other 8 arrays, the addition or
the product of these arrays makes up Array B, array A and array B are
then checked for correlation, we can then check to se if the rank
value breaks the current highest rank. Its basically a very simple
algorithm, i have already have a working version in VBA and it works
perfectly but the amount of time it would take to complete is is the
tens of years so remote psuedo threads seems the way forward i have
two models i have downloaded from sourceforge but i dont fully
understand how they work yet, hence the need for someone to help me,
and as i said i am willing to pay.