java based supercomputer

Discussion in 'Java' started by sav, Feb 14, 2007.

  1. sav

    sav Guest

    I am trying an ambitous project of trying to adapt a piece of code
    that i came accross on an IBM website. It uses psuedo remote threads
    and is a way of utilising RMI, multithreading and will develop a java
    based supercomuting technique. I wish to use a similar method for
    checking the correlation beteween an array of data and another array
    comprised of combining 8 other arrays of data, so by using a
    straightforward for next loop will take forever but by utilising this
    java psuedo remote threads will take a considerrably less time.
    Howeve, i am relatively new to java but i learn fast. I need someont
    to help me achieve this, i am willing to pay. I hacve the code
    available for review
     
    sav, Feb 14, 2007
    #1
    1. Advertising

  2. sav

    Daniel Pitts Guest

    On Feb 14, 1:56 pm, "sav" <> wrote:
    > I am trying an ambitous project of trying to adapt a piece of code
    > that i came accross on an IBM website. It uses psuedo remote threads
    > and is a way of utilising RMI, multithreading and will develop a java
    > based supercomuting technique. I wish to use a similar method for
    > checking the correlation beteween an array of data and another array
    > comprised of combining 8 other arrays of data, so by using a
    > straightforward for next loop will take forever but by utilising this
    > java psuedo remote threads will take a considerrably less time.
    > Howeve, i am relatively new to java but i learn fast. I need someont
    > to help me achieve this, i am willing to pay. I hacve the code
    > available for review


    This first question, which isn't even Java specific, is:
    Does your algorithm lend itself well to paralellization? You could
    find out by running it with normal Thread's on a multiprocessor
    system. If this improves performance, then you can proceed to the next
    questions.

    The second question, still not Java specific:
    When you split the computation between N threads, how much
    communication is there between those threads? If the communication
    between threads has higher latency or limited bandwidth, how does that
    effect your algorithms throughput?

    Java specific question: Have you profiled your existing "slow" program
    to see what is taking the most time? It might not be what you
    expect.

    Also, have you considered finding a less naive algorithm? Without
    more details about your goal I can only speculate, but it might be
    that your approach is an O(N^3) algorithm, when in fact there may
    exist an O(n*logn) algorithm.

    So, before trying to cross machine boundaries, or even thread
    boundaries, it might be worth looking into other algorithms that speed
    up your process.
     
    Daniel Pitts, Feb 14, 2007
    #2
    1. Advertising

  3. sav

    sav Guest

    On 14 Feb, 22:24, "Daniel Pitts" <> wrote:
    > On Feb 14, 1:56 pm, "sav" <> wrote:
    >
    > > I am trying an ambitous project of trying to adapt a piece of code
    > > that i came accross on an IBM website. It uses psuedo remote threads
    > > and is a way of utilising RMI, multithreading and will develop a java
    > > based supercomuting technique. I wish to use a similar method for
    > > checking the correlation beteween an array of data and another array
    > > comprised of combining 8 other arrays of data, so by using a
    > > straightforward for next loop will take forever but by utilising this
    > > java psuedo remote threads will take a considerrably less time.
    > > Howeve, i am relatively new to java but i learn fast. I need someont
    > > to help me achieve this, i am willing to pay. I hacve the code
    > > available for review

    >
    > This first question, which isn't even Java specific, is:
    > Does your algorithm lend itself well to paralellization? You could
    > find out by running it with normal Thread's on a multiprocessor
    > system. If this improves performance, then you can proceed to the next
    > questions.
    >
    > The second question, still not Java specific:
    > When you split the computation between N threads, how much
    > communication is there between those threads? If the communication
    > between threads has higher latency or limited bandwidth, how does that
    > effect your algorithms throughput?
    >
    > Java specific question: Have you profiled your existing "slow" program
    > to see what is taking the most time? It might not be what you
    > expect.
    >
    > Also, have you considered finding a less naive algorithm? Without
    > more details about your goal I can only speculate, but it might be
    > that your approach is an O(N^3) algorithm, when in fact there may
    > exist an O(n*logn) algorithm.
    >
    > So, before trying to cross machine boundaries, or even thread
    > boundaries, it might be worth looking into other algorithms that speed
    > up your process.


    Hi i did send you a full email to your email address did you get it?
     
    sav, Feb 15, 2007
    #3
  4. sav

    Daniel Pitts Guest

    On Feb 15, 11:16 am, "sav" <> wrote:
    > On 14 Feb, 22:24, "Daniel Pitts" <> wrote:
    >
    >
    >
    > > On Feb 14, 1:56 pm, "sav" <> wrote:

    >
    > > > I am trying an ambitous project of trying to adapt a piece of code
    > > > that i came accross on an IBM website. It uses psuedo remote threads
    > > > and is a way of utilising RMI, multithreading and will develop a java
    > > > based supercomuting technique. I wish to use a similar method for
    > > > checking the correlation beteween an array of data and another array
    > > > comprised of combining 8 other arrays of data, so by using a
    > > > straightforward for next loop will take forever but by utilising this
    > > > java psuedo remote threads will take a considerrably less time.
    > > > Howeve, i am relatively new to java but i learn fast. I need someont
    > > > to help me achieve this, i am willing to pay. I hacve the code
    > > > available for review

    >
    > > This first question, which isn't even Java specific, is:
    > > Does your algorithm lend itself well to paralellization? You could
    > > find out by running it with normal Thread's on a multiprocessor
    > > system. If this improves performance, then you can proceed to the next
    > > questions.

    >
    > > The second question, still not Java specific:
    > > When you split the computation between N threads, how much
    > > communication is there between those threads? If the communication
    > > between threads has higher latency or limited bandwidth, how does that
    > > effect your algorithms throughput?

    >
    > > Java specific question: Have you profiled your existing "slow" program
    > > to see what is taking the most time? It might not be what you
    > > expect.

    >
    > > Also, have you considered finding a less naive algorithm? Without
    > > more details about your goal I can only speculate, but it might be
    > > that your approach is an O(N^3) algorithm, when in fact there may
    > > exist an O(n*logn) algorithm.

    >
    > > So, before trying to cross machine boundaries, or even thread
    > > boundaries, it might be worth looking into other algorithms that speed
    > > up your process.

    >
    > Hi i did send you a full email to your email address did you get it?


    No. Three reasons. The first is that my home computer has been
    disabled for a few weeks.
    The second reason is that the given e-mail address gets so much spam
    that I usually just delete it with out looking.
    The third reason is that I didn't invite you to e-mail me, please keep
    the discussion here unless invited to otherwise.

    As far as I know, no one here gets paid to answer questions on this
    newsgroup. On the other hand, many of us DO get paid to answer
    questions one-on-one. If you want free advice, post here; if you want
    to pay for solutions, find a contractor.
     
    Daniel Pitts, Feb 15, 2007
    #4
  5. sav

    sav Guest

    On 15 Feb, 20:39, "Daniel Pitts" <> wrote:
    > On Feb 15, 11:16 am, "sav" <> wrote:
    >
    >
    >
    > > On 14 Feb, 22:24, "Daniel Pitts" <> wrote:

    >
    > > > On Feb 14, 1:56 pm, "sav" <> wrote:

    >
    > > > > I am trying an ambitous project of trying to adapt a piece of code
    > > > > that i came accross on an IBM website. It uses psuedo remote threads
    > > > > and is a way of utilising RMI, multithreading and will develop a java
    > > > > based supercomuting technique. I wish to use a similar method for
    > > > > checking the correlation beteween an array of data and another array
    > > > > comprised of combining 8 other arrays of data, so by using a
    > > > > straightforward for next loop will take forever but by utilising this
    > > > > java psuedo remote threads will take a considerrably less time.
    > > > > Howeve, i am relatively new to java but i learn fast. I need someont
    > > > > to help me achieve this, i am willing to pay. I hacve the code
    > > > > available for review

    >
    > > > This first question, which isn't even Java specific, is:
    > > > Does your algorithm lend itself well to paralellization? You could
    > > > find out by running it with normal Thread's on a multiprocessor
    > > > system. If this improves performance, then you can proceed to the next
    > > > questions.

    >
    > > > The second question, still not Java specific:
    > > > When you split the computation between N threads, how much
    > > > communication is there between those threads? If the communication
    > > > between threads has higher latency or limited bandwidth, how does that
    > > > effect your algorithms throughput?

    >
    > > > Java specific question: Have you profiled your existing "slow" program
    > > > to see what is taking the most time? It might not be what you
    > > > expect.

    >
    > > > Also, have you considered finding a less naive algorithm? Without
    > > > more details about your goal I can only speculate, but it might be
    > > > that your approach is an O(N^3) algorithm, when in fact there may
    > > > exist an O(n*logn) algorithm.

    >
    > > > So, before trying to cross machine boundaries, or even thread
    > > > boundaries, it might be worth looking into other algorithms that speed
    > > > up your process.

    >
    > > Hi i did send you a full email to your email address did you get it?

    >
    > No. Three reasons. The first is that my home computer has been
    > disabled for a few weeks.
    > The second reason is that the given e-mail address gets so much spam
    > that I usually just delete it with out looking.
    > The third reason is that I didn't invite you to e-mail me, please keep
    > the discussion here unless invited to otherwise.
    >
    > As far as I know, no one here gets paid to answer questions on this
    > newsgroup. On the other hand, many of us DO get paid to answer
    > questions one-on-one. If you want free advice, post here; if you want
    > to pay for solutions, find a contractor.


    ok cool, weel first to answer your questions, i am pretty sure that
    the algorithm will be quite easy to parralelise if that is the correct
    term, the only bottleneck i can see is checking the correlation value
    against the highest rank, ie the algorithm basically is a for next
    loop BUT with billions of calcs to perform, the bottlenck will
    obviously be when the calcs need to check for a higher correlation
    against the current highest rank, however with the java program i have
    seen it seesm to perform many of the loop functions simoultaneously,
    ie i have 8 arrays of of which perform 10^3 calculations so many of
    these calcs can be carried out concurrently.

    The current system runs in VBA not in java so i havent run in java
    yet.

    Look forward to your reply
     
    sav, Feb 15, 2007
    #5
  6. sav

    Daniel Pitts Guest

    On Feb 15, 12:54 pm, "sav" <> wrote:
    > On 15 Feb, 20:39, "Daniel Pitts" <> wrote:
    >
    >
    >
    > > On Feb 15, 11:16 am, "sav" <> wrote:

    >
    > > > On 14 Feb, 22:24, "Daniel Pitts" <> wrote:

    >
    > > > > On Feb 14, 1:56 pm, "sav" <> wrote:

    >
    > > > > > I am trying an ambitous project of trying to adapt a piece of code
    > > > > > that i came accross on an IBM website. It uses psuedo remote threads
    > > > > > and is a way of utilising RMI, multithreading and will develop a java
    > > > > > based supercomuting technique. I wish to use a similar method for
    > > > > > checking the correlation beteween an array of data and another array
    > > > > > comprised of combining 8 other arrays of data, so by using a
    > > > > > straightforward for next loop will take forever but by utilising this
    > > > > > java psuedo remote threads will take a considerrably less time.
    > > > > > Howeve, i am relatively new to java but i learn fast. I need someont
    > > > > > to help me achieve this, i am willing to pay. I hacve the code
    > > > > > available for review

    >
    > > > > This first question, which isn't even Java specific, is:
    > > > > Does your algorithm lend itself well to paralellization? You could
    > > > > find out by running it with normal Thread's on a multiprocessor
    > > > > system. If this improves performance, then you can proceed to the next
    > > > > questions.

    >
    > > > > The second question, still not Java specific:
    > > > > When you split the computation between N threads, how much
    > > > > communication is there between those threads? If the communication
    > > > > between threads has higher latency or limited bandwidth, how does that
    > > > > effect your algorithms throughput?

    >
    > > > > Java specific question: Have you profiled your existing "slow" program
    > > > > to see what is taking the most time? It might not be what you
    > > > > expect.

    >
    > > > > Also, have you considered finding a less naive algorithm? Without
    > > > > more details about your goal I can only speculate, but it might be
    > > > > that your approach is an O(N^3) algorithm, when in fact there may
    > > > > exist an O(n*logn) algorithm.

    >
    > > > > So, before trying to cross machine boundaries, or even thread
    > > > > boundaries, it might be worth looking into other algorithms that speed
    > > > > up your process.

    >
    > > > Hi i did send you a full email to your email address did you get it?

    >
    > > No. Three reasons. The first is that my home computer has been
    > > disabled for a few weeks.
    > > The second reason is that the given e-mail address gets so much spam
    > > that I usually just delete it with out looking.
    > > The third reason is that I didn't invite you to e-mail me, please keep
    > > the discussion here unless invited to otherwise.

    >
    > > As far as I know, no one here gets paid to answer questions on this
    > > newsgroup. On the other hand, many of us DO get paid to answer
    > > questions one-on-one. If you want free advice, post here; if you want
    > > to pay for solutions, find a contractor.

    >
    > ok cool, weel first to answer your questions, i am pretty sure that
    > the algorithm will be quite easy to parralelise if that is the correct
    > term, the only bottleneck i can see is checking the correlation value
    > against the highest rank, ie the algorithm basically is a for next
    > loop BUT with billions of calcs to perform, the bottlenck will
    > obviously be when the calcs need to check for a higher correlation
    > against the current highest rank, however with the java program i have
    > seen it seesm to perform many of the loop functions simoultaneously,
    > ie i have 8 arrays of of which perform 10^3 calculations so many of
    > these calcs can be carried out concurrently.
    >
    > The current system runs in VBA not in java so i havent run in java
    > yet.
    >
    > Look forward to your reply


    Okay, so if I understand correctly:

    You have N number of arrays to test;
    Testing array A doesn't interact with the testing of array B.
    Testing any array results in an easy-to-compare score.
    So, you could create up to N threads, each calculates and records the
    score.
    All of the scores can then be easily compared to one another.

    Since there are only 8 arrays (hence 8 scores), the sorting isn't too
    hard to do.

    How do you score array A? What do you mean by correlation? longest-
    common-subsequence? number of similar/same elements? Something else?
     
    Daniel Pitts, Feb 16, 2007
    #6
  7. sav

    sav Guest

    On 16 Feb, 00:17, "Daniel Pitts" <> wrote:
    > On Feb 15, 12:54 pm, "sav" <> wrote:
    >
    >
    >
    > > On 15 Feb, 20:39, "Daniel Pitts" <> wrote:

    >
    > > > On Feb 15, 11:16 am, "sav" <> wrote:

    >
    > > > > On 14 Feb, 22:24, "Daniel Pitts" <> wrote:

    >
    > > > > > On Feb 14, 1:56 pm, "sav" <> wrote:

    >
    > > > > > > I am trying an ambitous project of trying to adapt a piece of code
    > > > > > > that i came accross on an IBM website. It uses psuedo remote threads
    > > > > > > and is a way of utilising RMI, multithreading and will develop a java
    > > > > > > based supercomuting technique. I wish to use a similar method for
    > > > > > > checking the correlation beteween an array of data and another array
    > > > > > > comprised of combining 8 other arrays of data, so by using a
    > > > > > > straightforward for next loop will take forever but by utilising this
    > > > > > > java psuedo remote threads will take a considerrably less time.
    > > > > > > Howeve, i am relatively new to java but i learn fast. I need someont
    > > > > > > to help me achieve this, i am willing to pay. I hacve the code
    > > > > > > available for review

    >
    > > > > > This first question, which isn't even Java specific, is:
    > > > > > Does your algorithm lend itself well to paralellization? You could
    > > > > > find out by running it with normal Thread's on a multiprocessor
    > > > > > system. If this improves performance, then you can proceed to the next
    > > > > > questions.

    >
    > > > > > The second question, still not Java specific:
    > > > > > When you split the computation between N threads, how much
    > > > > > communication is there between those threads? If the communication
    > > > > > between threads has higher latency or limited bandwidth, how does that
    > > > > > effect your algorithms throughput?

    >
    > > > > > Java specific question: Have you profiled your existing "slow" program
    > > > > > to see what is taking the most time? It might not be what you
    > > > > > expect.

    >
    > > > > > Also, have you considered finding a less naive algorithm? Without
    > > > > > more details about your goal I can only speculate, but it might be
    > > > > > that your approach is an O(N^3) algorithm, when in fact there may
    > > > > > exist an O(n*logn) algorithm.

    >
    > > > > > So, before trying to cross machine boundaries, or even thread
    > > > > > boundaries, it might be worth looking into other algorithms that speed
    > > > > > up your process.

    >
    > > > > Hi i did send you a full email to your email address did you get it?

    >
    > > > No. Three reasons. The first is that my home computer has been
    > > > disabled for a few weeks.
    > > > The second reason is that the given e-mail address gets so much spam
    > > > that I usually just delete it with out looking.
    > > > The third reason is that I didn't invite you to e-mail me, please keep
    > > > the discussion here unless invited to otherwise.

    >
    > > > As far as I know, no one here gets paid to answer questions on this
    > > > newsgroup. On the other hand, many of us DO get paid to answer
    > > > questions one-on-one. If you want free advice, post here; if you want
    > > > to pay for solutions, find a contractor.

    >
    > > ok cool, weel first to answer your questions, i am pretty sure that
    > > the algorithm will be quite easy to parralelise if that is the correct
    > > term, the only bottleneck i can see is checking the correlation value
    > > against the highest rank, ie the algorithm basically is a for next
    > > loop BUT with billions of calcs to perform, the bottlenck will
    > > obviously be when the calcs need to check for a higher correlation
    > > against the current highest rank, however with the java program i have
    > > seen it seesm to perform many of the loop functions simoultaneously,
    > > ie i have 8 arrays of of which perform 10^3 calculations so many of
    > > these calcs can be carried out concurrently.

    >
    > > The current system runs in VBA not in java so i havent run in java
    > > yet.

    >
    > > Look forward to your reply

    >
    > Okay, so if I understand correctly:
    >
    > You have N number of arrays to test;
    > Testing array A doesn't interact with the testing of array B.
    > Testing any array results in an easy-to-compare score.
    > So, you could create up to N threads, each calculates and records the
    > score.
    > All of the scores can then be easily compared to one another.
    >
    > Since there are only 8 arrays (hence 8 scores), the sorting isn't too
    > hard to do.
    >
    > How do you score array A? What do you mean by correlation? longest-
    > common-subsequence? number of similar/same elements? Something else?


    Nok you are quite close its a little more compliacted than that but
    you have the basic idea, Array A values dont change they are fixed
    upon loaded from en excel file, i have written the code for that,
    array b is made up of values based upon the values from 8 other arrays
    each array is a calculation of (10^3)*N calcs, ie array 1 does a clac
    and is then added to the values of the other 8 arrays, the addition or
    the product of these arrays makes up Array B, array A and array B are
    then checked for correlation, we can then check to se if the rank
    value breaks the current highest rank. Its basically a very simple
    algorithm, i have already have a working version in VBA and it works
    perfectly but the amount of time it would take to complete is is the
    tens of years so remote psuedo threads seems the way forward i have
    two models i have downloaded from sourceforge but i dont fully
    understand how they work yet, hence the need for someone to help me,
    and as i said i am willing to pay.
     
    sav, Feb 16, 2007
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ken North
    Replies:
    0
    Views:
    379
    Ken North
    Jan 20, 2004
  2. Replies:
    1
    Views:
    382
  3. Replies:
    24
    Views:
    763
    Robin Becker
    Apr 15, 2005
  4. km
    Replies:
    1
    Views:
    392
    Christos TZOTZIOY Georgiou
    May 11, 2005
  5. sav

    supercomputer in java?

    sav, Feb 15, 2007, in forum: Java
    Replies:
    4
    Views:
    674
    Ravion
    Feb 16, 2007
Loading...

Share This Page