JDBC Threads Locking up

[sameergn]

Hi,

We are using Apache,Tomcat 5,ojdbc14.jar,Oracle 10g based system and
observing that all 100 tomcat threads are getting stuck in following
situation.

87 threads are stuck in "waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)"

TP-Processor100 daemon prio=1 tid=0x5c430080 nid=0x7964 waiting for
monitor entry [0x546c2000..0x546c3868]
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
129)
- waiting to lock <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x6254f030> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x6254f0f0> (a oracle.jdbc.driver.OracleConnection)

One thread has locked "<0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)" and it is waiting to
lock <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl).

TP-Processor45 daemon prio=1 tid=0x081cf9b8 nid=0x7d0 waiting for
monitor entry [0x562fa000..0x562fb868]
at
oracle.jdbc.pool.OracleConnectionCacheImpl.closeSingleConnection(OracleConnectionCacheImpl.java:
677)
- waiting to lock <0x718fc0e0> (a
oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.reusePooledConnection(OracleConnectionCacheImpl.java:
592)
at
oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleConnectionEventListener.java:
130)
- locked <0x718fc270> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:
482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:
445)
- locked <0x62354878> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:
2936)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:
1454)
- locked <0x62354938> (a oracle.jdbc.driver.OracleConnection)

One thread has locked <0x718fc0e0>, but it is waiting in
socketRead0(), so the lock can not released.
This stack trace is originating from while obtaining a new database
connection.

TP-Processor10 daemon prio=1 tid=0x084d4f50 nid=0x70d5 runnable
[0x5cd7c000..0x5cd7e868]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
t oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1894)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:
1094)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at
oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:
2132)
at
oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:
2015)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:
2877)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:
925)
- locked <0x62bc4228> (a oracle.jdbc.driver.OracleStatement)
- locked <0x62bb7a80> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.dbaccess.DBAccess.setNlsParamsClient(DBAccess.java:
1680)
at oracle.jdbc.ttc7.TTC7Protocol.initNls(TTC7Protocol.java:1592)
- locked <0x62bb7a10> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:
406)
at
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:
521)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:325)
at java.sql.DriverManager.getConnection(DriverManager.java:512)
- locked <0xa1efd890> (a java.lang.Class)
at java.sql.DriverManager.getConnection(DriverManager.java:140)
- locked <0xa1efd890> (a java.lang.Class)
at
oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:
171)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:
149)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
95)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:
63)
- locked <0x718fc1e0> (a
oracle.jdbc.pool.OracleConnectionPoolDataSource)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getNewPoolOrXAConnection(OracleConnectionCacheImpl.java:
547)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getPooledConnection(OracleConnectionCacheImpl.java:
404)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
298)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.getConnection(OracleConnectionCacheImpl.java:
268)
- locked <0x718fc0e0> (a oracle.jdbc.pool.OracleConnectionCacheImpl)

Did somebody ran into a similar situation? Please Help.

Thanks,
Sameer

 

[Juha Laiho]

(e-mail address removed) said:

We are using Apache,Tomcat 5,ojdbc14.jar,Oracle 10g based system and
observing that all 100 tomcat threads are getting stuck in following
situation.

That seems familiar (though I don't have exact thread dump available
from the issue I was solving a while ago).

What you need is a patched jdbc driver. The tough thing is, this isn't
readily downloadable from f.ex. Oracle TechNet. So, you'll need:
- Oracle support contract (and Oracle MetaLink account)
- Oracle 10g installation that you're allowed to patch
.... search for JDBC patches to your particular 10g version,
and apply them to your Oracle installation. Then copy over the jdbc
driver that now contains the patches.

I would supply you with the patch number, if I just had it, but I don't
have access to it any longer.