Strange RMI failure

[John C. Bollinger]

Hi folks,

I wondering whether anyone can give me a lead on a strange RMI failure.
After successfully retrieving a remote object stub from the RMI
server, the client receives a java.rmi.ConnectException when it tries to
invoke a method (via the stub) on the remote object. The exception
specifies "refused to host: 127.0.0.1", and the root exception is a
java.net.ConnectException thrown by the RMI stack when it tries to open
a socket. I know the stub was retrieved because its class appears a few
levels down in the stack trace. I don't have the full stack trace
accessible at the moment but I'll provide it as soon as I can if the
above doesn't clue anyone in.

An identical copy of the client worked successfully against a different
host running the same version of the RMI server, but the JVM versions
on the server are not identical (Sun 1.4.2 vs. Sun 1.4.2_03). There may
be configuration differences, but I'm not sure what to look for. The
server and client software were compiled together with javac of Sun
1.4.2_03. The servers are running on RedHat Linux 8 with the Sun Java
packages, and the clients are running in Windows 2000/XP.

If this sounds familiar to anyone, or if there are any suggestions as to
what I should check then I would be very pleased to hear about it.

Cheers,

John Bollinger
(e-mail address removed)

 

[Sudsy]

John C. Bollinger wrote:

If this sounds familiar to anyone, or if there are any suggestions as to
what I should check then I would be very pleased to hear about it.

John,
It sounds as though the naming service only knows about the loopback
interface, not the "external" IP of the server. Your client (on a
different machine) is trying to connect to itself (the local loopback)
since that's the address being returned from the lookup.
At least that would be my guess based on the provided information.
I could well be wrong...

 

[Tony Morris]

Hi folks,

I wondering whether anyone can give me a lead on a strange RMI failure.
After successfully retrieving a remote object stub from the RMI
server, the client receives a java.rmi.ConnectException when it tries to
invoke a method (via the stub) on the remote object. The exception
specifies "refused to host: 127.0.0.1", and the root exception is a
java.net.ConnectException thrown by the RMI stack when it tries to open
a socket. I know the stub was retrieved because its class appears a few
levels down in the stack trace. I don't have the full stack trace
accessible at the moment but I'll provide it as soon as I can if the
above doesn't clue anyone in.

An identical copy of the client worked successfully against a different
host running the same version of the RMI server, but the JVM versions
on the server are not identical (Sun 1.4.2 vs. Sun 1.4.2_03). There may
be configuration differences, but I'm not sure what to look for. The
server and client software were compiled together with javac of Sun
1.4.2_03. The servers are running on RedHat Linux 8 with the Sun Java
packages, and the clients are running in Windows 2000/XP.

If this sounds familiar to anyone, or if there are any suggestions as to
what I should check then I would be very pleased to hear about it.

Cheers,

John Bollinger
(e-mail address removed)

The RMI server is binding to the loopback address.
You might want to use java.rmi.registry.LocateRegistry

--
Tony Morris
(BInfTech, Cert 3 I.T.)
Software Engineer
IBM Australia - Tivoli Security Software
(2003 VTR1000F)
Sun Certified Programmer for the Java 2 Platform (1.4)
Sun Certified Developer for the Java 2 Platform

 

[John C. Bollinger]

It sounds as though the naming service only knows about the loopback
interface, not the "external" IP of the server. Your client (on a
different machine) is trying to connect to itself (the local loopback)
since that's the address being returned from the lookup.
At least that would be my guess based on the provided information.
I could well be wrong...

Thanks for the comment. Yes, I saw the loopback address, but I didn't
know what to make of it because the remote stub was successfully
retrieved. That is, I took the successful retrieval of the stub to mean
that the naming service worked correctly. That may have been an
inappropriate assumption. If I understand correctly what Tony Morris
said, on the other hand, it may make sense of the whole thing. Stay tuned.


Cheers,

John Bollinger
(e-mail address removed)

 

[John C. Bollinger]

The RMI server is binding to the loopback address.
You might want to use java.rmi.registry.LocateRegistry

Thanks, Tony, that got me pointed in the right direction. I am still
working out the details, but I found that if I specified the
java.rmi.server.hostname system property to the server's VM, thereby
supplying the server's IP address on the local ethernet network instead
of the loopback address, then the problem went away. The problem seems
quite similar to Java bug 4460816, which involves the setup of the
systems /etc/hosts file, and indeed the hosts file on the problematic
server is a bit odd.

What appears to be happening is that the RMI registry and naming lookup
is fine, but the RMI server is configuring the stub instances with the
loopback address as the address of IP of the originating VM. Thus the
client hasn't any problem obtaining the stubs, but it cannot use them
because in its context they don't point at the correct location for the
remote object. Moreover, RMI works fine between client and server on
the same machine host, because in that case the loopback address goes to
the same place for both. That last masked the problem for us until
quite recently.

When I finish characterizing the configuration details that cause this
behavior I'll update the group. I may also post a bug report with Sun,
but I'm not confident right now that it should really be characterized
as a Java bug.


John Bollinger
(e-mail address removed)

 

[John C. Bollinger]

When I finish characterizing the configuration details that cause this
behavior I'll update the group. I may also post a bug report with Sun,
but I'm not confident right now that it should really be characterized
as a Java bug.

I have completed my analysis to my own satisfaction. The problem
manifested when both of the following two criteria were satisfied:

(1) The server machine's /etc/hosts file did not contain an entry for
the machine's own network IP, and

(2) The JVM in which the RMI server application was running did not have
an appropriate value of the java.rmi.system.hostname system property
specified to it (including if no value was specified).

It appears that in the absence of a host name specified via
java.rmi.system.hostname, the RMI server uses the hosts file to
determine to which address(es) it should bind exported objects. I do
not consider this behavior a JVM bug, but I do wish I had been able to
find some documentation of it.

And as an aside, I had somewhat of a fight over item (1) above with the
sysadmin whom I employ to manage this and related systems. I maintained
that it is both conventional and good practice that every machine should
specify its own name(s) and address(es) in its hosts file, particularly
if the resolver is configured to use the file; he argued that doing so
duplicates information whose primary location (for RedHat 8) is
elsewhere, thus complicating management of the system. This isn't
really the right forum for it, but I'd be happy to hear the opinions of
those around here on the question.

Cheers,

John Bollinger

 

[Sudsy]

John C. Bollinger wrote:

And as an aside, I had somewhat of a fight over item (1) above with the
sysadmin whom I employ to manage this and related systems. I maintained
that it is both conventional and good practice that every machine should
specify its own name(s) and address(es) in its hosts file, particularly
if the resolver is configured to use the file; he argued that doing so
duplicates information whose primary location (for RedHat 8) is
elsewhere, thus complicating management of the system. This isn't
really the right forum for it, but I'd be happy to hear the opinions of
those around here on the question.

I'll reply, though there's no guarantee that you'll see this: your
newsreader appears to ignore posts from the "free mail" services.
Since you have to hardcode both the IP address and hostname in
order that your "run command" (rc) files can properly initialize
the interfaces, how much more effort is involved in setting these
values in /etc/hosts?
My take? Your hosts file should include all your local interfaces
(including loopback) as well as your gateway(s) and DNS server(s)
names and addresses. Does it take a lot of additional management
effort? Only if you change address frequently, something which
shouldn't be happening IMHO.

 

[Esmond Pitt]

It appears that in the absence of a host name specified via

java.rmi.system.hostname, the RMI server uses the hosts file to
determine to which address(es) it should bind exported objects. I do
not consider this behavior a JVM bug, but I do wish I had been able to
find some documentation of it.

No, this is not a JVM bug. It is the correct behaviour of what the JVM
calls to find out, i.e. gethostname()/gethostbyname() in the BSD sockets
API. These consult DNS if available and configured as available,
otherwise the 'hosts' file. Your sysadmin is partially correct in that
if the information is available in DNS it shouldn't be repeated in the
'hosts' file, but clearly it wasn't found in DNS either, so there is
some misconfiguration somewhere. The server host must have a way of
discovering its own host name, and then the external IP address of that
hostname, from either DNS or the 'hosts' file or, in the case of RMI,
java.rmi.server.hostname, otherwise RMI uses 127.0.0.1, which doesn't
export in stubs to other hosts successfully.

Esmond Pitt