Distributed Java Programming Model

H

howa

Hi,

Are there any recommended Java Programming Model or Framework that is
suitable for writing something like that:

1. A master program control many workers, each workers will perform
some cpu intensive computation

2. Therefore, it is perferred these workers run on remote PC

3. The master take care of monitoring, logging, and could restart the
worker if any problem, e.g. worker's process being killed

4. Finally, all the workers send result to master, and ask for other
jobs
 
M

Marc E

I do a lot of this, but not #3 with restarting. I use RMI.

so a farm of 10 servers are the workers, and each has RMI started, waiting
to accept requests. the master then sends requests via RMI and gets the
result. if the RMI invocation fails because the server isn't accepting
connections, it notifies people so they can check out that worker.

I use the spring framework for doing all this which makes the RMI stuff as
trivial as calling a method on the local machine. The only thing you have to
do is make sure the result that the RMI workers send back is a serializable
object, i.e. just add "implements Serializable" to your result object and
it's all good.
 
C

Chris Smith

howa said:
Are there any recommended Java Programming Model or Framework that is
suitable for writing something like that:

Are you looking for something that already exists? As Marc mentioned,
RMI and serialization provides a way to move the work around without a
lot of low-level network details. The rest of it, though, is not part
of RMI. Building the rest of the system is an interesting idea, but I
haven't seen it yet.

Incidentally, despite Marc's comment, I don't think Spring buys you
anything at all here; it looks like Marc is attributing to Spring
features of the basic RMI setup. I wouldn't add a framework like Spring
unless you have a real need for its features, and it's generally not
designed for this kind of endeavor.

In any case, if you were to make yourself familiar with RMI, building
something to handle the logging, monitoring, and coordination could be a
nice weekend task; it shouldn't be prohibitive.
 
L

Lew

Marc said:
The only thing you have to
do is make sure the result that the RMI workers send back is a serializable
object, i.e. just add "implements Serializable" to your result object and
it's all good.

After reading Joshua Bloch's _Effective Java_, wherein he deeply attacks the
myth of "just add 'implements Serializable'", I am skeptical. Making a class
Serializable for the long haul (he does say the simple method works for
throwaway classes) requires careful design.

The book explains it better than I can, but you need a serialVersionUid static
member, object serialization and deserialization routines, invariant
guarantors and more.

None of this matters if different versions of the class need not be
compatible, which makes it valid to "just add 'implements Serializable'" after
all. In that circumstance.

- Lew
 
M

Marc E

Very true, Chris. Never having worked with RMI before, and having the
project already using Spring, it was nice to just add a few lines of XML to
the context file and have the RMI plumbing handled for me
 
H

howa

Chris Smith ¼g¹D¡G
Are you looking for something that already exists? As Marc mentioned,
RMI and serialization provides a way to move the work around without a
lot of low-level network details. The rest of it, though, is not part
of RMI. Building the rest of the system is an interesting idea, but I
haven't seen it yet.

Incidentally, despite Marc's comment, I don't think Spring buys you
anything at all here; it looks like Marc is attributing to Spring
features of the basic RMI setup. I wouldn't add a framework like Spring
unless you have a real need for its features, and it's generally not
designed for this kind of endeavor.

In any case, if you were to make yourself familiar with RMI, building
something to handle the logging, monitoring, and coordination could be a
nice weekend task; it shouldn't be prohibitive.

I am looking for some library or framework, such as IBM's DPPEJ.

I want to expore more in direction, since the underlying detail of
monitoring, load balacing can be hidden from the application layer.
 
C

Chris Uppal

howa said:
1. A master program control many workers, each workers will perform
some cpu intensive computation

2. Therefore, it is perferred these workers run on remote PC

3. The master take care of monitoring, logging, and could restart the
worker if any problem, e.g. worker's process being killed

4. Finally, all the workers send result to master, and ask for other
jobs

Have you looked into JavaSpaces ?

(Most, perhaps all, of the material on the Web about JavaSpaces is laden with
idiotic "enterprisey" manager-oriented double-speak, if you can ignore that
then you should find that the underlying reality is quite simple, and /may/
provide the features you are looking for.)

-- chris
 
P

Patrick May

howa said:
Are there any recommended Java Programming Model or Framework that is
suitable for writing something like that:

1. A master program control many workers, each workers will perform
some cpu intensive computation

2. Therefore, it is perferred these workers run on remote PC

3. The master take care of monitoring, logging, and could restart the
worker if any problem, e.g. worker's process being killed

4. Finally, all the workers send result to master, and ask for other
jobs

As Chris Uppal has already noted, this is an ideal application
for a JavaSpace. The master-worker pattern is at the heart of many
uses of JavaSpaces. Here's an example:

http://www.gigaspaces.com/wiki/display/GS/Compute+Grid

Regards,

Patrick
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,902
Latest member
Elena68X5

Latest Threads

Top