RMI & connection refused

[Duane Evenson]

I'm suffering from the the common "java.rmi.ConnectException: Connection
refused to host" problem.

I've searched the web and found this problem described many times.
However, the solutions, range from changing /etc/hosts to doing really
weird stuff -- but nothing applicable.

All the online RMI tutorials and examples work fine when you're using the
same host, but as soon as the client moves to a remote location, all the
examples fail.

This is how it is with my code as well.

Here is an example with some simple code:
::::::::::::::
RMIClient.java
::::::::::::::
import java.rmi.registry.*;

public class RMIClient {
public static void main(String[] args) {
String host = args[0];
int port = Integer.parseInt(args[1]);
if (System.getSecurityManager() == null)
System.setSecurityManager(new SecurityManager());
try {
Registry r = LocateRegistry.getRegistry(host, port);
ServerIface s = (ServerIface) r.lookup("RMIServerName");
System.out.println(s.hello());
} catch (Exception e) {
e.printStackTrace();
}
}
}
::::::::::::::
RMIServer.java
::::::::::::::
import java.rmi.RemoteException;
import java.rmi.registry.*;
import java.rmi.server.UnicastRemoteObject;

public class RMIServer extends UnicastRemoteObject implements ServerIface {
protected RMIServer() throws RemoteException { super(); }
public static void main(String[] args) {
String host = args[0];
int port = Integer.parseInt(args[1]);
if (System.getSecurityManager() == null)
System.setSecurityManager(new SecurityManager());
try {
Registry r = LocateRegistry.getRegistry(host, port);
r.bind("RMIServerName", new RMIServer());
} catch (Exception e) {
e.printStackTrace();
}
}
public String hello() throws RemoteException {
return "Hello World";
}
}
::::::::::::::
ServerIface.java
::::::::::::::
import java.rmi.Remote;
import java.rmi.RemoteException;

public interface ServerIface extends Remote {
public abstract String hello() throws RemoteException;
}

I also have a wide open security policy file:
::::::::::::::
policy
::::::::::::::
grant {
permission java.security.AllPermission;
};

Here's my commands:

rmiregistry &
java -Djava.security.policy=policy RMIServer localhost 1099 &
java -Djava.security.policy=policy RMIClient localhost 1099

The problem is with the next command from a remote host:

java -Djava.security.policy=policy RMIClient adam 1099
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:179)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:132)
at $Proxy0.hello(Unknown Source)
at RMIClient.main(RMIClient.java:12)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
....


I tried switching server and clients, (and had to change ports) but
without any success.

I think the registry is OK, I make a simple ListRMI program using
Naming.list() to view the registry entries.

Can anyone help -- not just for me -- but for all the rest looking for a
definitive answer to what's going on and how to fix it?
Thanks!

 

[Tom Anderson]

Here's my commands:

rmiregistry &
java -Djava.security.policy=policy RMIServer localhost 1099 &
java -Djava.security.policy=policy RMIClient localhost 1099

The problem is with the next command from a remote host:

java -Djava.security.policy=policy RMIClient adam 1099
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused

Why is it connecting to 127.0.0.1? That's the local loopback interface -
you know about that, right? If the client and server are on different
machines, that doesn't seem like a totally likely to work thing to do.
What's your /etc/hosts look like? What to you get if you do 'ping adam'?

A simple but good diagnostic tool in situations like this is telnet. On
the client machine, telnet to port 1099 on the server machine. If you
manage to make a connection, you know that there's nothing wrong at the
network level, and any problem must be in the program. If you can't make a
connection, you know there's trouble with the network - firewalls, DNS,
routing, something like that.

tom

 

[Karl Uppiano]

I'm suffering from the the common "java.rmi.ConnectException: Connection
refused to host" problem.
[...]

Can anyone help -- not just for me -- but for all the rest looking for a
definitive answer to what's going on and how to fix it?
Thanks!

Java RMI is not particularly firewall-friendly. If you are running Windows,
you need to open ports to allow RMI to pass. Note that the RMI registry is
one service, and each object you publish is another service, accepting
connections on arbitrarily assigned ports (by default). These arbitrarily
assigned ports are a problem for firewall admins, because you don't know in
advance which port the objects will be listening on.

One of the reasons for the RMI Registry is to give you a one-stop-shop for
your arbitrarily assigned service ports, but it doesn't solve the firewall
problem. You can create a custom RMI socket factory that will create server
sockets on specifically assigned ports.
http://java.sun.com/j2se/1.4.2/docs/api/java/rmi/server/RMISocketFactory.html

If you are running Windows, temporarily try turning off the Windows firewall
on both the RMI client and RMI server. If it works, then you have identified
the problem (no, not the firewall...).

 

[Duane Evenson]

Why is it connecting to 127.0.0.1? That's the local loopback interface -
you know about that, right? If the client and server are on different
machines, that doesn't seem like a totally likely to work thing to do.
What's your /etc/hosts look like? What to you get if you do 'ping adam'?

A simple but good diagnostic tool in situations like this is telnet. On
the client machine, telnet to port 1099 on the server machine. If you
manage to make a connection, you know that there's nothing wrong at the
network level, and any problem must be in the program. If you can't make a
connection, you know there's trouble with the network - firewalls, DNS,
routing, something like that.

tom

I assume that reference to 127.0.0.1 isn't important in tracing out the
problem. It is just how the registry refers to the server -- after all
they are the only two who are on the same server, if the client knows the
IP of the registry, it knows the IP of the server.

I think I know the problem here. I assumed the server would keep the port
open with which it communicate with the registry. Then, at the network
level, would use "Related" state to switch to communicating with the
client. Further reading of the Javadoc for UnicastRemoteObject finds
that the server listens on a second, random port for a client. I don't
know how most firewalls are supposed to handle this, but I used a
different constructor that specified a fixed, non-firewalled port.

Now I'm getting another problem -- when I execute the server, I'm getting
errors, but the server still continues (hangs like it should) but nothing
is registered with the RMI name server.

I'm getting:
Server exception: java.rmi.ServerException: RemoteException occurred in
server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested
exception is: java.lang.ClassNotFoundException: dbServer.Server
java.rmi.ServerException: RemoteException occurred in server thread;
nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested
exception is: java.lang.ClassNotFoundException: dbServer.Server at
sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:385)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:240)
at sun.rmi.transport.Transport$1.run(Transport.java:153) at
java.security.AccessController.doPrivileged(Native Method) at
sun.rmi.transport.Transport.serviceCall(Transport.java:149)
....

Using the -verbose flag, I can see that dbServer.Server (which is the
interface for the server is loading OK:

....
[Loaded java.security.cert.Certificate from /usr/local/jdk1.5.0_06/jre/lib/rt.ja
r]
[Loaded java.rmi.Remote from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded dbServer.Server from file:/home/duanee/]
[Loaded java.rmi.server.RemoteObject from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.rmi.server.RemoteServer from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.rmi.server.UnicastRemoteObject from /usr/local/jdk1.5.0_06/jre/lib/
rt.jar]
[Loaded dbServer.ServerImpl from file:/home/duanee/]
[Loaded java.io.IOException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.rmi.RemoteException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.sql.SQLException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[...

I'm running Java 1.5 so I don't need stubs. Still, I tried precompiling
them with rmic, but the problem persists.

 

[Nigel Wade]

I assume that reference to 127.0.0.1 isn't important in tracing out the
problem.

It is very relevant. It's the host your client is trying to connect to.

It is just how the registry refers to the server --

The Naming.list() call to the rmiregistry returns the list of registered RMI
servers prefixed by the interface on which the request was made. If the client
makes a request to the rmiregistry on host 1.2.3.4 to get a list of RMI servers
using Naming.list("//1.2.3.4/") then the list of RMI servers returned by would
be //1.2.3.4:1099/RMIserver1, //1.2.3.4:1099/RMIserver2 etc. The registry list
includes the registry host, but not the server IP.

after all
they are the only two who are on the same server, if the client knows the
IP of the registry, it knows the IP of the server.

Not at all. There may be only one rmiregistry for the subnet, but there may be
several RMI servers each registering with that registry. The rmiregistry lookup
(e.g. Naming.lookup("//1.2.3.4:1099/RMIserver1")) should return a Remote object
already connected to the RMI server in question. The RMI server may or may not
be on host 1.2.3.4.

I think I know the problem here. I assumed the server would keep the port
open with which it communicate with the registry. Then, at the network
level, would use "Related" state to switch to communicating with the
client. Further reading of the Javadoc for UnicastRemoteObject finds
that the server listens on a second, random port for a client. I don't
know how most firewalls are supposed to handle this, but I used a
different constructor that specified a fixed, non-firewalled port.

That is one of the joys of this type of networking. Usually you have to open all
high ports (1024 up) incoming from the client. I imagine that RMI has any built
in way of opening ports (it would hardly be portable, and Java should be
portable). Offhand I can't remember whether RMI allows a server to listen on a
specific port and then register that port with rmiregistry. If it did at least
you'd only have to open one port.

Now I'm getting another problem -- when I execute the server, I'm getting
errors, but the server still continues (hangs like it should) but nothing
is registered with the RMI name server.

I'm getting:
Server exception: java.rmi.ServerException: RemoteException occurred in
server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested
exception is: java.lang.ClassNotFoundException: dbServer.Server
java.rmi.ServerException: RemoteException occurred in server thread;
nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested
exception is: java.lang.ClassNotFoundException: dbServer.Server at
sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:385)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:240)
at sun.rmi.transport.Transport$1.run(Transport.java:153) at
java.security.AccessController.doPrivileged(Native Method) at
sun.rmi.transport.Transport.serviceCall(Transport.java:149)
...

IIRC this happens if the rmiregisty, RMI server and/or the RMI client are using
different versions of the same class, one of the classes which needs to be
marshalled/unmarshalled as a RMI call parameter or return value. Marshalling
relies on the serialVersionUID to ensure the class is consistent on both client
and server. Check the versions of the classes used in each instance.
Recompiling (unless you have correctly used your own serialVersionUID) will
result in a new UID being assigned to the class and that will prevent
marshalling if a different version of the class is used on the client/server.
If you recompile on the server then you need to propagate the new class files
to the rmiregistry and the client.

Using the -verbose flag, I can see that dbServer.Server (which is the
interface for the server is loading OK:

...
[Loaded java.security.cert.Certificate

from /usr/local/jdk1.5.0_06/jre/lib/rt.ja

r]
[Loaded java.rmi.Remote from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded dbServer.Server from file:/home/duanee/]
[Loaded java.rmi.server.RemoteObject

from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]

[Loaded java.rmi.server.RemoteServer

from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]

[Loaded java.rmi.server.UnicastRemoteObject

from /usr/local/jdk1.5.0_06/jre/lib/

rt.jar]
[Loaded dbServer.ServerImpl from file:/home/duanee/]
[Loaded java.io.IOException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.rmi.RemoteException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[Loaded java.sql.SQLException from /usr/local/jdk1.5.0_06/jre/lib/rt.jar]
[...

I'm running Java 1.5 so I don't need stubs. Still, I tried precompiling
them with rmic, but the problem persists.

 

[Nigel Wade]

You can use RMI without a registry and declare the port your service is
accessed on and you can open just that one in your firewall.

Ok, I wasn't aware of that. I've only used RMI with the rmiregisty to handle
port numbers for the client. I generally only use RMI on the one host
(client=server=registry), and only have ports open to my development machine
and blocked to all other systems. I couldn't see anything in the RMI API which
allowed you to specify the RMI server port number.

If your server
happens to create additional remote objects, each will get its own port. You
don't have to know or handle the port number, but a firewall will block
access. Similarly, remote objects instantiated on the client (RMI
callbacks) will open a server socket on the client and the server's remote
proxy will attempt to connect to that port. Again, you don't have to know
any of the port numbers for this to work, I'd rather not open ports on the
client.

If you don't use the rmiregistry, and you don't know or handle the port numbers,
how does the client determine the port number on which the server is listening?

 

[Roedy Green]

Can anyone help -- not just for me -- but for all the rest looking for a
definitive answer to what's going on and how to fix it?

Some things to try:

RMI likes everything everywhere to be the exact same version,
including the application code.

Is your firewall at both ends letting the RMI traffic through in both
directions? Typically you must configure your firewall to redirect
incoming requests to the appropriate server on the LAN.
see http://mindprod.com/bgloss/firewall.html

If you run Wireshark on both server and client (if possible) can you
see requests getting through? see
http://mindprod.com/jgloss/wireshark.html

You might experiment with a miniature echoserver. See
http://mindprod.com/products2.html#ECHOSERVER configured to use the
same ports as RMI just to see if you can get socket traffic through.

are you using any sort of authentication of the client to permit
access to the stubs on the server?

If you are using SSL, turn it off temporarily till you get things
working, then turn it back on.
--
Roedy Green Canadian Mind Products
http://mindprod.com

"Patriotism is fierce as a fever, pitiless as the grave, blind as a stone, and as irrational as a headless hen."
~ Ambrose Bierce (born: 1842-06-24 died: 1914 at age: 71)

 

[EJP]

You can use RMI without a registry and declare the port your service is
accessed on and you can open just that one in your firewall.

No you can't. You can only get an RMI remote reference (stub) as the
result of a remote method invocation, which constitutes a bootstrap
problem. The only two ways around it are (i) have the server serialize
its stub to a file on a shared directory and have the client read that
(or similarly use any other sharable resource) or (ii) the RMI Registry,
which incorporates a bootstrap mechanism.

@OP: the solution to your problem is item A.1 on the RMI Frequently
Asked Questions list in the Javadoc.

 

[Nigel Wade]

If you don't use the rmiregistry, and you don't know or handle the port
numbers,
how does the client determine the port number on which the server is
listening?

Sorry, I was unclear. Clients need the port number to find the top-level
server object. Remote objects instantiated within the server get dynamic
port numbers, but the client doesn't need to know them (see below). For
other instantiated remote objects and RMI callback objects, the port numbers
are managed automatically. The code below shows the basic structure (but is
otherwise uncompiled, untested, unstructured, etc.) In using Eclipse w/ Java
1.6 I don't need to invoke rmic--stubs and skeletons are handled
automatically. Just make sure all the class referenced by the client are
actually available to it. In the example below, that would be Server and
Session. ServerImpl and SessionImpl are not needed by the client.
[snip]
public static final void main (String [] args) {
LocateRegistry.createRegistry(5005 /* your port number */);
ServerImpl actualServer = new ServerImpl ();
Naming.rebind("YourServerName", actualServer );
// Wait until directed to exit
}
}

public class ClientApp {
public static final void main (String [] args) {
Server remoteServer = Naming.lookup ("//<host>:5005/YourServerName");
Session newSession = remoteServer.createSession ("my", "pw");
}
}

But surely the above code is using an RMI registry (albeit, an internal one).
Rather than register with an external rmiregistry process it creates its own
with LocateRegistry.createRegistry. The client still needs to know the port
number on which the server started this registry. The registry port and the
dynamic ports used by the registered services still need to be open in the
firewall to the client, and given service ports are dynamic in nature that
still most likely requires opening up all ports >= 1024.

From the client perspective surely this is identical to using rmiregisty, and
isn't that the entire point? All the client knows or cares is that some
registry is running on the host/port which it has been given.

 

[EJP]

I think what you're saying is that the stub includes the dynamic port number.

The stub includes the dynamic port number.

Would you think

that because the stub and instance are produced automatically (I don't use
rmic and there are no stub / skeleton classfiles), it seems that if the
client should be able to produce the stub itself and fill-in the externally
declared host and port number?

No. The client has no way to learn the dynamic port number, and even if
it did it still doesn't know the other stuff in the stub such as the
UUID of the remote object.

The RMI Registry and the Activation system provide the only practical
bootstraps.

(i) basically means using a shared directory + serialization instead of
the Registry. It's not all that practical, and given that the RMI
Registry exists why would you use anything else?

 

[Nigel Wade]

The question I was answering was how to manage the port numbers so one need
open only those on the firewall, which is complicated with dynamic ports.

I am pretty sure that the design of RMI and the way it uses dynamic ports dates
back to the use of dynamic ports in ONC/RPC (which was also developed by Sun
for NFS).

The idea behind using dynamic ports is that it gets over some limitations
associated with static ports. The first problem is whether to use a low
numbered port (<1024) or a high port. Low ports carry with them some additional
security because they can normally only be opened by applications with elevated
permissions, e.g. root on UNIX/Linux or administrator rights on Windows. This
means the the client can be more trusting of the server, but the server must be
started by a superuser account. There are also a very limited number of low
ports, and unless you get your port number accepted by IANA as a reserved port
there's a good chance you'll find some other service camped on your port.

Alternatively, if you try to use a fixed high numbered port you are into the
realms of hope - hope that the port you want to use hasn't already been
assigned as a random port to some other process. Generally there is no way to
reserve high numbered ports for your application, you just have to try to open
the port and hope for the best.

The second problem with static ports is that if the server crashes there's a
good chance that the port will end up in the TIMED_WAIT state and the server
won't be able to restart because the port is still in use. You may be able to
get around this if you can set the SO_REUSEADDR socket option, but that will
just open another can of worms.

So, ideally what you want is to be able use dynamic, high-numbered ports. But
how does your client know where your server is? This is where
portmapper/rmiregistry come in. When the server starts, and is assigned its
random port, it registers itself with portmapper/rmiregistry saying what
service it offers and on what port that service is available. When the client
wants to avail itself of a service it contacts portmapper/rmiregistry and
discovers where that service is located.

This setup allows you to run servers which don't need to be run by an account
with elevated privilege (hence reducing the security implications), listening
on any available port, and all for the cost of one single fixed low port
number. It's your proverbial win-win situation.

And then along came firewalls...

However, the stub's port number does not have to be dynamic (RMI FAQ C.11).
The server can assign the remote object's port number to a pre-defined
value.

Unfortunately this does re-introduces one of the problems with fixed ports.
Namely the server crashing and the port still being in use so the server is
unable to restart.

The registry is still needed--I was mistaken about being able to
export the stub easily without it. Two ports would need to be opened--one
for the registry and one for the server. Just as with dynamic ports, the
client would not need to be aware of the server's port number--it needs only
the registry number. I'm testing a sample project that shows how to do
this. It's pretty easy, but if there are additional issues I havn't
considered, please let me know.

Just that one point above. If the server crashes it may not be able to restart
on the same port. Most of the time you'll be fine, but occasionally the server
will refuse to restart and you'll wonder what's gone wrong.