Distributed Java Programming Model

Discussion in 'Java' started by howa, Dec 10, 2006.

  1. howa

    howa Guest

    Hi,

    Are there any recommended Java Programming Model or Framework that is
    suitable for writing something like that:

    1. A master program control many workers, each workers will perform
    some cpu intensive computation

    2. Therefore, it is perferred these workers run on remote PC

    3. The master take care of monitoring, logging, and could restart the
    worker if any problem, e.g. worker's process being killed

    4. Finally, all the workers send result to master, and ask for other
    jobs
    howa, Dec 10, 2006
    #1
    1. Advertising

  2. howa

    Marc E Guest

    I do a lot of this, but not #3 with restarting. I use RMI.

    so a farm of 10 servers are the workers, and each has RMI started, waiting
    to accept requests. the master then sends requests via RMI and gets the
    result. if the RMI invocation fails because the server isn't accepting
    connections, it notifies people so they can check out that worker.

    I use the spring framework for doing all this which makes the RMI stuff as
    trivial as calling a method on the local machine. The only thing you have to
    do is make sure the result that the RMI workers send back is a serializable
    object, i.e. just add "implements Serializable" to your result object and
    it's all good.




    "howa" <> wrote in message
    news:...
    > Hi,
    >
    > Are there any recommended Java Programming Model or Framework that is
    > suitable for writing something like that:
    >
    > 1. A master program control many workers, each workers will perform
    > some cpu intensive computation
    >
    > 2. Therefore, it is perferred these workers run on remote PC
    >
    > 3. The master take care of monitoring, logging, and could restart the
    > worker if any problem, e.g. worker's process being killed
    >
    > 4. Finally, all the workers send result to master, and ask for other
    > jobs
    >
    Marc E, Dec 10, 2006
    #2
    1. Advertising

  3. howa

    Chris Smith Guest

    howa <> wrote:
    > Are there any recommended Java Programming Model or Framework that is
    > suitable for writing something like that:


    Are you looking for something that already exists? As Marc mentioned,
    RMI and serialization provides a way to move the work around without a
    lot of low-level network details. The rest of it, though, is not part
    of RMI. Building the rest of the system is an interesting idea, but I
    haven't seen it yet.

    Incidentally, despite Marc's comment, I don't think Spring buys you
    anything at all here; it looks like Marc is attributing to Spring
    features of the basic RMI setup. I wouldn't add a framework like Spring
    unless you have a real need for its features, and it's generally not
    designed for this kind of endeavor.

    In any case, if you were to make yourself familiar with RMI, building
    something to handle the logging, monitoring, and coordination could be a
    nice weekend task; it shouldn't be prohibitive.

    --
    Chris Smith
    Chris Smith, Dec 10, 2006
    #3
  4. howa

    Lew Guest

    Marc E wrote:
    > The only thing you have to
    > do is make sure the result that the RMI workers send back is a serializable
    > object, i.e. just add "implements Serializable" to your result object and
    > it's all good.


    After reading Joshua Bloch's _Effective Java_, wherein he deeply attacks the
    myth of "just add 'implements Serializable'", I am skeptical. Making a class
    Serializable for the long haul (he does say the simple method works for
    throwaway classes) requires careful design.

    The book explains it better than I can, but you need a serialVersionUid static
    member, object serialization and deserialization routines, invariant
    guarantors and more.

    None of this matters if different versions of the class need not be
    compatible, which makes it valid to "just add 'implements Serializable'" after
    all. In that circumstance.

    - Lew
    Lew, Dec 11, 2006
    #4
  5. howa

    Marc E Guest

    Very true, Chris. Never having worked with RMI before, and having the
    project already using Spring, it was nice to just add a few lines of XML to
    the context file and have the RMI plumbing handled for me

    "Chris Smith" <> wrote in message
    news:...
    > howa <> wrote:
    >> Are there any recommended Java Programming Model or Framework that is
    >> suitable for writing something like that:

    >
    > Are you looking for something that already exists? As Marc mentioned,
    > RMI and serialization provides a way to move the work around without a
    > lot of low-level network details. The rest of it, though, is not part
    > of RMI. Building the rest of the system is an interesting idea, but I
    > haven't seen it yet.
    >
    > Incidentally, despite Marc's comment, I don't think Spring buys you
    > anything at all here; it looks like Marc is attributing to Spring
    > features of the basic RMI setup. I wouldn't add a framework like Spring
    > unless you have a real need for its features, and it's generally not
    > designed for this kind of endeavor.
    >
    > In any case, if you were to make yourself familiar with RMI, building
    > something to handle the logging, monitoring, and coordination could be a
    > nice weekend task; it shouldn't be prohibitive.
    >
    > --
    > Chris Smith
    Marc E, Dec 11, 2006
    #5
  6. howa

    howa Guest

    Chris Smith ¼g¹D¡G

    > howa <> wrote:
    > > Are there any recommended Java Programming Model or Framework that is
    > > suitable for writing something like that:

    >
    > Are you looking for something that already exists? As Marc mentioned,
    > RMI and serialization provides a way to move the work around without a
    > lot of low-level network details. The rest of it, though, is not part
    > of RMI. Building the rest of the system is an interesting idea, but I
    > haven't seen it yet.
    >
    > Incidentally, despite Marc's comment, I don't think Spring buys you
    > anything at all here; it looks like Marc is attributing to Spring
    > features of the basic RMI setup. I wouldn't add a framework like Spring
    > unless you have a real need for its features, and it's generally not
    > designed for this kind of endeavor.
    >
    > In any case, if you were to make yourself familiar with RMI, building
    > something to handle the logging, monitoring, and coordination could be a
    > nice weekend task; it shouldn't be prohibitive.
    >
    > --
    > Chris Smith


    I am looking for some library or framework, such as IBM's DPPEJ.

    I want to expore more in direction, since the underlying detail of
    monitoring, load balacing can be hidden from the application layer.
    howa, Dec 11, 2006
    #6
  7. howa

    Chris Uppal Guest

    howa wrote:

    > 1. A master program control many workers, each workers will perform
    > some cpu intensive computation
    >
    > 2. Therefore, it is perferred these workers run on remote PC
    >
    > 3. The master take care of monitoring, logging, and could restart the
    > worker if any problem, e.g. worker's process being killed
    >
    > 4. Finally, all the workers send result to master, and ask for other
    > jobs


    Have you looked into JavaSpaces ?

    (Most, perhaps all, of the material on the Web about JavaSpaces is laden with
    idiotic "enterprisey" manager-oriented double-speak, if you can ignore that
    then you should find that the underlying reality is quite simple, and /may/
    provide the features you are looking for.)

    -- chris
    Chris Uppal, Dec 11, 2006
    #7
  8. howa

    Patrick May Guest

    "howa" <> writes:
    > Are there any recommended Java Programming Model or Framework that is
    > suitable for writing something like that:
    >
    > 1. A master program control many workers, each workers will perform
    > some cpu intensive computation
    >
    > 2. Therefore, it is perferred these workers run on remote PC
    >
    > 3. The master take care of monitoring, logging, and could restart the
    > worker if any problem, e.g. worker's process being killed
    >
    > 4. Finally, all the workers send result to master, and ask for other
    > jobs


    As Chris Uppal has already noted, this is an ideal application
    for a JavaSpace. The master-worker pattern is at the heart of many
    uses of JavaSpaces. Here's an example:

    http://www.gigaspaces.com/wiki/display/GS/Compute Grid

    Regards,

    Patrick

    ------------------------------------------------------------------------
    S P Engineering, Inc. | Large scale, mission-critical, distributed OO
    | systems design and implementation.
    | (C++, Java, Common Lisp, Jini, middleware, SOA)
    Patrick May, Dec 13, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Riad Djemili

    Survey on distributed pair programming

    Riad Djemili, Jul 2, 2006, in forum: Java
    Replies:
    0
    Views:
    314
    Riad Djemili
    Jul 2, 2006
  2. Maciej Sobczak
    Replies:
    0
    Views:
    292
    Maciej Sobczak
    Jul 8, 2003
  3. Replies:
    1
    Views:
    474
    Andreas Wollschlaeger
    Oct 6, 2006
  4. Replies:
    2
    Views:
    330
    mlimber
    Feb 3, 2006
  5. howa
    Replies:
    0
    Views:
    291
Loading...

Share This Page