DatagramSocket.send problem on Linux 2.6 with Jre 1.4.2

[suresh.ru]

Hi,

We are running Linux 2.6.9-22.EL with Jre 1.4.2 on a AMD-64 machine. We
have an application that opens a DatagramSocket connection and sends a
packet. The application works fine as standalone application. But, when
invoked through a servlet, there is unsatisfied link error from
libnio.so for libnet.so, whereas libnet.so is available in the PATH
that the exception shows up.

Also, when we replace the Jre with 1.5 everything works fine. So, we
are confused... :-(

Any insight into this problem will be highly appreciated.

Thanks,
Suresh R

 

[rajesh]

can you paste the error here? I think some library files not found
while running your application.

 

[Nigel Wade]

Hi,

We are running Linux 2.6.9-22.EL with Jre 1.4.2 on a AMD-64 machine. We
have an application that opens a DatagramSocket connection and sends a
packet. The application works fine as standalone application. But, when
invoked through a servlet, there is unsatisfied link error from
libnio.so for libnet.so, whereas libnet.so is available in the PATH
that the exception shows up.

Also, when we replace the Jre with 1.5 everything works fine. So, we
are confused... :-(

Any insight into this problem will be highly appreciated.

Thanks,
Suresh R

If the problem is only in the servlet then it's probably something wrong with
the configuration of the servlet engine. What engine are you using?

I use Tomcat5 on a very similar setup, 2.6.9-22.EL, Opteron, and jre 1.4.2_11.
The servlets I am running on this setup are part of the AstroGrid
(http://www.astrtrogrid.org.uk) project, so make heavy use of networking code.
There is no issue here with unsatisfied external links for libnet.so.

 

[suresh.ru]

HI,

This is exactly the same configuration that I am also using (my jre is
1.4.2_8, everything else is just same).

I tried with Jre1.5 and it works. So, if it is configuration problem
then should'nt work for 1.5 too, right?

I tried to set the java.library.path so that the library gets loaded
and the library load problem went. But, results in the following
calss-def-not-found exception with no details on the class, though :-(

Exception trace for java.lang.NoClassDefFoundError:

java.lang.NoClassDefFoundError
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(Unknown
Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
...

Suresh R

 

[Nigel Wade]

(e-mail address removed) wrote:

Please, don't top-post. It makes a total mess of the thread.
I've moved your reply to the bottom.

HI,

This is exactly the same configuration that I am also using (my jre is
1.4.2_8, everything else is just same).

I tried with Jre1.5 and it works. So, if it is configuration problem
then should'nt work for 1.5 too, right?

Impossible to say. There are configuration errors which could easily make it
work with one version and not with another. The libraries used by the JVM are
not linked in such a way that they can be found at runtime without additional
help. The JVM normally knows where the libraries are located, but if something
has got messed up with the libraries or the installation directories they won't
be found.

However, there is one thing which I have just remembered. I use the 32bit 1.4
SDK rather than the 64bit one. This might have some bearing on the issue. Which
version do you run?

 

[suresh.ru]

Please, don't top-post. It makes a total mess of the thread.

I've moved your reply to the bottom.

Sorry about that. I hope this one does'nt show up like that.

Impossible to say. There are configuration errors which could easily make it
work with one version and not with another. The libraries used by the JVM are
not linked in such a way that they can be found at runtime without additional
help. The JVM normally knows where the libraries are located, but if something
has got messed up with the libraries or the installation directories they won't
be found.

I tried adding the library path explicity in catalina.sh in the
CATALINA_OPTS using -Djava.library.path. Also, I moved all the shared
libraries to $jre-home/lib/ext/. But, none of these seems to have an
effect :-(

Just to be sure that there has been no mix and match of the libraries,
I downloaded and used the the latest Jre that I could find from Sun's
site, ie,

/usr/java/j2re1.4.2_12/bin/java -version
java version "1.4.2_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_12-b03)
Java HotSpot(TM) Client VM (build 1.4.2_12-b03, mixed mode)

and I problem still persits... :-(

OS: 2.6.9-22.EL #1 Mon Sep 19 17:49:49 EDT 2005 x86_64 x86_64 x86_64
GNU/Linux

However, there is one thing which I have just remembered. I use the 32bit 1.4
SDK rather than the 64bit one. This might have some bearing on the issue. Which
version do you run?

readelf -h /usr/java/j2re1.4.2_12/bin/java
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Intel 80386
Version: 0x1
Entry point address: 0x8049180
Start of program headers: 52 (bytes into file)
Start of section headers: 50152 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 6
Size of section headers: 40 (bytes)
Number of section headers: 26
Section header string table index: 25

Its 32-bit version. So, still no luck.

Following is the sample program I was using to test
1) Application: SockTest.java
import java.net.InetAddress;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.logging.Level;
import java.nio.channels.SocketChannel;

class SockTest {
public static void main(String[] args) {
System.out.println("shr> BEGIN sock test...");
try {
SocketChannel.open().close();
System.out.println("shr> SUCCESS!!");
} catch (Exception e) {
System.out.println("shr> FAILED :-(");
e.printStackTrace();
}
System.out.println("shr> END sock test...");
}
}

2) Servlet version of the same: SockServletTest.java
import java.io.*;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import java.net.InetAddress;
import java.nio.channels.SocketChannel;

public class SockServletTest extends HttpServlet
{
protected void service(HttpServletRequest request,
HttpServletResponse response)
throws ServletException, IOException
{
System.out.println("shr> BEGIN sock test...");
try {
SocketChannel.open().close();
System.out.println("shr> SUCCESS!!");
} catch (Exception e) {
System.out.println("shr> FAILED :-(");
e.printStackTrace();
}
System.out.println("shr> END sock test...");
}
}

One interesting observation is that when I first invoke the above
servlet, I get the following UnsatisfiedLink error. But, after
subsequent calls, I get NoClassDefFound exception.

Exception during first time call:
2006-09-25 13:34:22 StandardWrapperValve[sockTest]: Servlet.service()
for servlet sockTest threw exception
java.lang.UnsatisfiedLinkError:
/opt/OV/nonOV/jre/1.4/lib/i386/libnio.so: libnet.so: cannot open shared
object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at sun.security.action.LoadLibraryAction.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.Util.load(Unknown Source)
at sun.nio.ch.SocketChannelImpl.<clinit>(Unknown Source)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(Unknown
Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
at SockServletTest.service(SockServletTest.java:18)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Unknown Source)

Exception during subsequent calls:
2006-09-25 13:35:28 StandardWrapperValve[sockTest]: Servlet.service()
for servlet sockTest threw exception
java.lang.NoClassDefFoundError
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(Unknown
Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
at SockServletTest.service(SockServletTest.java:18)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Unknown Source)


Any leads would be really helpful.

Thanks,
Suresh R

 

[Gordon Beaton]

One interesting observation is that when I first invoke the above
servlet, I get the following UnsatisfiedLink error. But, after
subsequent calls, I get NoClassDefFound exception.

Exception during first time call:
2006-09-25 13:34:22 StandardWrapperValve[sockTest]: Servlet.service()
for servlet sockTest threw exception
java.lang.UnsatisfiedLinkError:
/opt/OV/nonOV/jre/1.4/lib/i386/libnio.so: libnet.so: cannot open shared
object file: No such file or directory

I'll assume that you've confirmed that the library really is there...

If that's the case, then libnio.so is itself dependent on other
libraries (and so on, recursively), at least one of which could not be
loaded, which ultimately causes the load to fail. Use ldd or readelf
-d to see what libraries libnio.so depends on.

Note that java.library.path only helps the JVM to find the libraries
it loads directly. Libraries that are further down the dependency
chain must be where ld.so can find them, so you need to either set
LD_LIBRARY_PATH, or edit /etc/ld.so.conf (and run ldconfig to update
the cache).

You've mentioned that your code works from a regular application but
fails in a servlet. I don't know about servlets, but if there are
classloaders involved, then your problems might be caused by
attempting to load the necessary native libraries from one
classloader, and subsequently attempting to load them a second time
from another classloader, or use them from a classloader where they
aren't visible.

/gordon

 

[suresh.ru]

I'll assume that you've confirmed that the library really is there...

Yes, thats correct. The libraries are present.

If that's the case, then libnio.so is itself dependent on other
libraries (and so on, recursively), at least one of which could not be
loaded, which ultimately causes the load to fail. Use ldd or readelf
-d to see what libraries libnio.so depends on.

In fact, I did a explicit System.load of the dependent libraries in a
static block. But that lead to exception somewhere else saying that
some of them were already loaded (this was tried from the actual
program, not by using the one I had posted in this thread).

Note that java.library.path only helps the JVM to find the libraries
it loads directly. Libraries that are further down the dependency
chain must be where ld.so can find them, so you need to either set
LD_LIBRARY_PATH, or edit /etc/ld.so.conf (and run ldconfig to update
the cache).

Setting LD_LIBRARY_PATH works in the shell where I set it. I start the
tomcat server from the same shell, but I still see the problem. I also
tried adding the directories in the ld.so.conf. But that did'nt help
either.

One thing worked partially. I copied the whole of '*.so' from jre/lib,
jre/lib/i386/ jre/lib/i386/server, jre/lib/i386/client to /usr/lib and
the exception went away. Again, it did'nt work fully, as it failed
later in the socket send.

You've mentioned that your code works from a regular application but
fails in a servlet. I don't know about servlets, but if there are
classloaders involved, then your problems might be caused by
attempting to load the necessary native libraries from one
classloader, and subsequently attempting to load them a second time
from another classloader, or use them from a classloader where they
aren't visible.

I have little knowledge on the classloaders. But, I dont understand as
to why, from context of the same JVM, would the classloader be not able
to recognise if some piece of code was not already loaded. I am
expecting it work pretty much like a loader. A loader knows if its
already available in memory and loads accordingly for us, right... on
the same lines, I mean. I will go through some materials on
classloader, understand how it impacts our code and try to see if we
can do something about it.

Suresh R

 

[Gordon Beaton]

I have little knowledge on the classloaders. But, I dont understand as
to why, from context of the same JVM, would the classloader be not able
to recognise if some piece of code was not already loaded.

There is a hierarchy of classloaders, and AFAIK code loaded in one
classloader is only visible in the same and descendent classloaders
(i.e. below it in the tree).

Similarly, when a native library is loaded in one classloader, it is
only visible to classes loaded in the same and subordinate
classloaders. At the same time, the JVM *prevents* the same shared
library from being loaded multiple times (even in different
classloaders), which creates a catch 22 situation. It's long since I
looked at this kind of problem, and things may have changed by now.

Admittedly it's strange that this should affect the inbuilt classes
though.

These may help:
http://www.unix.org.ua/orelly/java-ent/servlet/ch13_05.htm
http://forum.java.sun.com/thread.jspa?threadID=440954&messageID=1996591

/gordon