JVM crash while loading native dll using TOMCAT

[Madhav]

hi all,
I am facing a strange problem with tomcat and jvm. I load a
JNI class which loads a native dll and this native dll creates a com+
object. Amazing thing is that this works when the tomcat server starts
from the command line ( ie going to tomcat bin dir and running tomcat
5..exe with no options) but it fails when started from the system tray
icon. I got the following error while doing so:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x05af8510, pid=4200,
tid=2332
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_06-b05 mixed mode,
sharing)
# Problematic frame:
# C [ImageAdapter.dll+0x8510]
#

--------------- T H R E A D ---------------

Current thread (0x03321f80): JavaThread "TaskThread" daemon
[_thread_in_native, id=2332]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

Registers:
EAX=0x00000000, EBX=0x05af13d7, ECX=0x00000000, EDX=0x05b12d68
ESP=0x057ded74, EBP=0x057ded7c, ESI=0x057df0a8, EDI=0x057df778
EIP=0x05af8510, EFLAGS=0x00010246

Top of Stack: (sp=0x057ded74)
0x057ded74: 05af401c 00000000 057dee04 05af60f2
0x057ded84: 00000000 057df778 057df0a8 05af13d7
0x057ded94: 00000001 0000000c 0000000c 05af9040
0x057deda4: 0000000c 05af906b 0000000c 00000000
0x057dedb4: 05af67fe 0000000c 057df0a8 05b12d68
0x057dedc4: 057df778 057df0a8 05af13d7 05b12d00
0x057dedd4: 05b12d68 057df4f4 057dedc4 057dee48
0x057dede4: 05b021a0 ffffffff 057dedfc 00000000

Instructions: (pc=0x05af8510)
0x05af8500: 00 00 8d a4 24 00 00 00 00 8d a4 24 00 00 00 00
0x05af8510: 8b 01 ba ff fe fe 7e 03 d0 83 f0 ff 33 c2 83 c1


Stack: [0x056e0000,0x057e0000), sp=0x057ded74, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
C [ImageAdapter.dll+0x8510]
C [ImageAdapter.dll+0x60f2]
C [ImageAdapter.dll+0x141a]
j
com.waveset.adapter.ImageAdapter.getAllUsers(Ljava/lang/StringLjava/lang/String;+0
j
com.waveset.adapter.ImageAdapterResource$TableAccountIterator.<init>(Lcom/waveset/adapter/ImageAdapterResource;Ljava/lang/StringV+84
j
com.waveset.adapter.ImageAdapterResource.getAccountIterator()Lcom/waveset/adapter/AccountIterator;+49
j
com.waveset.adapter.ResourceAdapterProxy.getAccountIterator()Lcom/waveset/adapter/AccountIterator;+20
j
com.waveset.recon.ImportTask.extractAccountObjects(Lcom/waveset/util/BlockingQueue;Lcom/waveset/object/ObjectCache;Lcom/waveset/object/Extract;Lcom/waveset/task/Monitor;Lcom/waveset/object/Subject;[Z[Ljava/lang/ThrowableI+319
j
com.waveset.recon.ImportTask.doRealWork(Lcom/waveset/task/TaskContext;Lcom/waveset/object/TaskInstanceV+144
j
com.waveset.task.Executor.execute(Lcom/waveset/task/TaskContext;Lcom/waveset/object/TaskInstanceV+151
j com.waveset.task.TaskThread.run()V+146
v ~StubRoutines::call_stub
V [jvm.dll+0x845a9]
V [jvm.dll+0xd9317]
V [jvm.dll+0x8447a]
V [jvm.dll+0x841d7]
V [jvm.dll+0x9ed69]
V [jvm.dll+0x109fe3]
V [jvm.dll+0x109fb1]
C [msvcrt.dll+0x291ed]
C [kernel32.dll+0xa990]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j
com.waveset.adapter.ImageAdapter.getAllUsers(Ljava/lang/StringLjava/lang/String;+0
j
com.waveset.adapter.ImageAdapterResource$TableAccountIterator.<init>(Lcom/waveset/adapter/ImageAdapterResource;Ljava/lang/StringV+84
j
com.waveset.adapter.ImageAdapterResource.getAccountIterator()Lcom/waveset/adapter/AccountIterator;+49
j
com.waveset.adapter.ResourceAdapterProxy.getAccountIterator()Lcom/waveset/adapter/AccountIterator;+20
j
com.waveset.recon.ImportTask.extractAccountObjects(Lcom/waveset/util/BlockingQueue;Lcom/waveset/object/ObjectCache;Lcom/waveset/object/Extract;Lcom/waveset/task/Monitor;Lcom/waveset/object/Subject;[Z[Ljava/lang/ThrowableI+319
j
com.waveset.recon.ImportTask.doRealWork(Lcom/waveset/task/TaskContext;Lcom/waveset/object/TaskInstanceV+144
j
com.waveset.task.Executor.execute(Lcom/waveset/task/TaskContext;Lcom/waveset/object/TaskInstanceV+151
j com.waveset.task.TaskThread.run()V+146
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x033e1af8 JavaThread "TraceLogger" daemon [_thread_blocked, id=4104]
0x033f1d00 JavaThread "Thread-14" daemon [_thread_blocked, id=4136]
=>0x03321f80 JavaThread "TaskThread" daemon [_thread_in_native,
id=2332]
0x033f7b20 JavaThread "Audit Log Handler" daemon [_thread_blocked,
id=3368]
0x03322718 JavaThread "http-8080-3" daemon [_thread_blocked, id=5908]
0x0330bb68 JavaThread "http-8080-2" daemon [_thread_blocked, id=5836]
0x03363de8 JavaThread "Reconciler Work Thread" daemon
[_thread_blocked, id=4428]
0x0337b0c0 JavaThread "Reconciler Work Thread" daemon
[_thread_blocked, id=4484]
0x034a56b0 JavaThread "Reconciler Resource Thread" daemon
[_thread_blocked, id=4436]
0x037f0f78 JavaThread "Source Adapter Task madhav" daemon
[_thread_blocked, id=4452]
0x0379ce70 JavaThread "Reconciler" daemon [_thread_blocked, id=4468]
0x0372c780 JavaThread "ajp-8009-1" daemon [_thread_blocked, id=4420]
0x03498cd8 JavaThread "ajp-8009-Poller" daemon [_thread_blocked,
id=4404]
0x03498b50 JavaThread "ajp-8009-Acceptor" daemon [_thread_in_native,
id=4388]
0x0372a010 JavaThread "http-8080-1" daemon [_thread_blocked, id=4372]
0x032dcce0 JavaThread "http-8080-Sendfile" daemon [_thread_blocked,
id=4356]
0x0361b638 JavaThread "http-8080-Poller" daemon [_thread_in_native,
id=4340]
0x0361b418 JavaThread "http-8080-Acceptor" daemon [_thread_in_native,
id=4292]
0x03792e80 JavaThread
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon
[_thread_blocked, id=4324]
0x036bec18 JavaThread "Task Scheduler" daemon [_thread_blocked,
id=4276]
0x03555ae8 JavaThread "Scheduler.PollThread" daemon [_thread_blocked,
id=4260]
0x035fb0f8 JavaThread "ChangeNotifier update thread" daemon
[_thread_blocked, id=4244]
0x037e3240 JavaThread "AuthorizationTypes watcher thread" daemon
[_thread_blocked, id=4228]
0x034c14a8 JavaThread "System Configuration Watcher Thread" daemon
[_thread_blocked, id=4212]
0x008b72b0 JavaThread "Thread-1" [_thread_in_native, id=4172]
0x008591f0 JavaThread "Low Memory Detector" daemon [_thread_blocked,
id=4124]
0x00857ee8 JavaThread "CompilerThread0" daemon [_thread_blocked,
id=4108]
0x008570e0 JavaThread "Signal Dispatcher" daemon [_thread_blocked,
id=4272]
0x003ef868 JavaThread "Finalizer" daemon [_thread_blocked, id=4256]
0x00851aa0 JavaThread "Reference Handler" daemon [_thread_blocked,
id=4248]
0x003e6e60 JavaThread "main" [_thread_in_native, id=4176]

Other Threads:
0x003ee2a8 VMThread [id=4240]
0x00857020 WatcherThread [id=4140]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 18240K, used 17183K [0x06a70000,
0x07e30000, 0x091d0000)
eden space 16256K, 93% used [0x06a70000, 0x07947db0, 0x07a50000)
from space 1984K, 100% used [0x07a50000, 0x07c40000, 0x07c40000)
to space 1984K, 0% used [0x07c40000, 0x07c40000, 0x07e30000)
tenured generation total 241984K, used 151010K [0x091d0000,
0x17e20000, 0x26a70000)
the space 241984K, 62% used [0x091d0000, 0x12548908, 0x12548a00,
0x17e20000)
compacting perm gen total 13824K, used 13624K [0x26a70000,
0x277f0000, 0x2aa70000)
the space 13824K, 98% used [0x26a70000, 0x277be0f8, 0x277be200,
0x277f0000)
ro space 8192K, 63% used [0x2aa70000, 0x2af7b178, 0x2af7b200,
0x2b270000)
rw space 12288K, 46% used [0x2b270000, 0x2b809fa8, 0x2b80a000,
0x2be70000)

Dynamic libraries:
0x00400000 - 0x0041b000 C:\Program Files\Apache Software
Foundation\Tomcat 5.5\bin\tomcat5.exe
0x77f40000 - 0x77ffa000 C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f34000 C:\WINDOWS\system32\kernel32.dll
0x77d00000 - 0x77d8f000 C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c44000 C:\WINDOWS\system32\GDI32.dll
0x77da0000 - 0x77e30000 C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cf4000 C:\WINDOWS\system32\RPCRT4.dll
0x77290000 - 0x772d9000 C:\WINDOWS\system32\SHLWAPI.dll
0x77ba0000 - 0x77bf4000 C:\WINDOWS\system32\msvcrt.dll
0x77380000 - 0x77b5d000 C:\WINDOWS\system32\SHELL32.dll
0x70ad0000 - 0x70bb6000
C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.100.0_x-ww_8417450B\comctl32.dll
0x6d670000 - 0x6d804000 C:\Program
Files\Java\jre1.5.0_06\bin\client\jvm.dll
0x76aa0000 - 0x76acc000 C:\WINDOWS\system32\WINMM.dll
0x6d280000 - 0x6d288000 C:\Program Files\Java\jre1.5.0_06\bin\hpi.dll
0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
0x76f50000 - 0x76f63000 C:\WINDOWS\system32\Secur32.dll
0x6d640000 - 0x6d64c000 C:\Program
Files\Java\jre1.5.0_06\bin\verify.dll
0x6d300000 - 0x6d31d000 C:\Program Files\Java\jre1.5.0_06\bin\java.dll
0x6d660000 - 0x6d66f000 C:\Program Files\Java\jre1.5.0_06\bin\zip.dll
0x03810000 - 0x038e5000 C:\Program Files\Apache Software
Foundation\Tomcat 5.5\bin\tcnative-1.dll
0x71c00000 - 0x71c18000 C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b63000 C:\WINDOWS\system32\MSWSOCK.dll
0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
0x03970000 - 0x0399d000 C:\WINDOWS\system32\rsaenh.dll
0x6d4c0000 - 0x6d4d3000 C:\Program Files\Java\jre1.5.0_06\bin\net.dll
0x76ed0000 - 0x76ef7000 C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3f000 C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000 C:\WINDOWS\system32\rasadhlp.dll
0x6d000000 - 0x6d167000 C:\Program Files\Java\jre1.5.0_06\bin\awt.dll
0x73070000 - 0x73096000 C:\WINDOWS\system32\WINSPOOL.DRV
0x76290000 - 0x762ad000 C:\WINDOWS\system32\IMM32.dll
0x77160000 - 0x77284000 C:\WINDOWS\system32\ole32.dll
0x05af0000 - 0x05b0d000 C:\Program Files\Apache Software
Foundation\Tomcat 5.5\shared\lib\ImageAdapter.dll
0x770e0000 - 0x7715d000 C:\WINDOWS\system32\OLEAUT32.dll
0x76f90000 - 0x7700e000 C:\WINDOWS\system32\CLBCatQ.DLL
0x77010000 - 0x770d6000 C:\WINDOWS\system32\COMRes.dll
0x77b90000 - 0x77b98000 C:\WINDOWS\system32\VERSION.dll

VM Arguments:
jvm_args: -Dcatalina.home=C:\Program Files\Apache Software
Foundation\Tomcat 5.5 -Dcatalina.base=C:\Program Files\Apache Software
Foundation\Tomcat 5.5 -Djava.endorsed.dirs=C:\Program Files\Apache
Software Foundation\Tomcat 5.5\common\endorsed
-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat
5.5\temp
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.util.logging.config.file=C:\Program Files\Apache Software
Foundation\Tomcat 5.5\conf\logging.properties vfprintf -Xms256m
-Xmx512m -Xss1000k
java_command: <unknown>
Launcher Type: generic

Environment Variables:
PATH=C:\Program Files\CA\DCS\CAWIN\;C:\Program Files\CA\Advantage
Ingres [ET]\ingres\bin;C:\Program Files\CA\Advantage Ingres
[ET]\ingres\utility;C:\oraclexe\app\oracle\product\10.2.0\server\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program
Files\CA\SharedComponents\CAM\bin;C:\Program
Files\CA\SharedComponents\CAM\bin;C:\PROGRA~1\CA\SHARED~1\ETRUST~1\Bin;C:\PROGRA~1\Tcl\bin;C:\Program
Files\CA\eTrust Directory\dxserver\bin;C:\Program
Files\CA\SharedComponents\CA_APPSW\;C:\Program Files\CA\eTrust
Admin\bin;C:\Program Files\CA\SharedComponents\CAM\bin;C:\Program
Files\Microsoft SQL Server\80\Tools\BINN;C:\Program
Files\WinRAR;;C:\PROGRA~1\CA\ET4BEC~1\Bin;C:\Program Files\Apache
Software Foundation\Tomcat 5.5\shared\lib
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 3, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790

CPU:total 2 family 15, cmov, cx8, fxsr, mmx, sse, sse2, ht

Memory: 4k page, physical 1039084k(133180k free), swap 2505712k(915776k
free)

vm_info: Java HotSpot(TM) Client VM (1.5.0_06-b05) for windows-x86,
built on Nov 10 2005 11:12:14 by "java_re" with MS VC++ 6.0

 

[Gordon Beaton]

I am facing a strange problem with tomcat and jvm. I load a JNI
class which loads a native dll and this native dll creates a com+
object. Amazing thing is that this works when the tomcat server
starts from the command line ( ie going to tomcat bin dir and
running tomcat 5..exe with no options) but it fails when started
from the system tray icon. I got the following error while doing so:

There may well be known bugs in your infrastructure, but the very
first thing to suspect is your own native code.

/gordon

 

[Madhav]

There may well be known bugs in your infrastructure, but the very

first thing to suspect is your own native code.

Unfortunatly, the Com+ code is not accessible. All I have done is write
a c++ wrapper over it and create a stub dll. Following is my native
code :

_bstr_t xmlResponse="";
CoInitialize (NULL);
of<<"After CoInit"<<endl;
try {
of<<"Before IMC Domain ptr"<<endl;
IMCDomain2Ptr md("ESMC.MCDomain");
of<<"After IMC Domain ptr"<<endl;
//md->DomainName = "BCTEST2";
md->DomainName = domainStr;
of<<"After domain set"<<endl;

The jvm crashes just after "Before IMC domain ptr" message occurs.
And when the tomcat starts from command line, it works fine.

 

[Chris Uppal]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000

That looks like an unchecked null-pointer access in the native code. Since you
say that this is third party code, probably your only option is to get onto
them and have them solve the problem.

-- chris

 

[Madhav]

That looks like an unchecked null-pointer access in the native code. Since you
say that this is third party code, probably your only option is to get onto
them and have them solve the problem.

-- chris

Actually this code ran perfectly well when tomcat was started from
commandline without any options. It fails only when tomcat starts as a
service.

 

[Gordon Beaton]

Actually this code ran perfectly well when tomcat was started from
commandline without any options. It fails only when tomcat starts as a
service.

So there are bugs in the code, but you have yet to see their effect
when you run it without options from the command line.

/gordon

 

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

Actually this code ran perfectly well when tomcat was started from
commandline without any options. It fails only when tomcat starts as a
service.

Username, access, environment ?

Arne

 

[travishein]

Also having issues with native .dll when tomcat is started in service mode.

I found this post interesting, as I was having similar problems when I was trying to get Tomcat 6, using Java 1.6 on a windows platform to run a web application that made use of a Java-COM bridge to communicate with Quickbooks. using v7 of their SDK.

I found that when I ran the connector Java program in stand-alone mode from a .bat file and a main() method I was able to connect and do stuff. But when I bundled it into a web application and launched it from inside tomcat, It would fail with an error message about not being able to connect. I did not see a visible crash message, but that may not mean there was not a crash, perhaps the output was absorbed somewhere into a log file. I had tried a lot of things, like moving the .dll files into the windows\system32 folder to make sure they were on the native library path for tomcat use. But still was not able to run it from inside tomcat.

After reading this post I realized that I was always starting tomcat as a service, either from the control panel services, or the task bar tomcat starter applet. So I shut down the tomcat service and started it up from the batch file instead, and the stuff using the native code in the Quickbooks Java-COM bridge now works.

So there must be something that is different in the system environment variables or settings used to launch the java vm that tomcat runs in, between the how it is done from the catalina.bat and how it is done from the service manager. ?

 

[travishein]

Ok, I think I found the difference, posting a reply to my own post for the completeness.

When tomcat is started as a windows service from the service control manager, it is by default started with the local system account. For some reason, this local system user is not able to invoke COM objects for other applications, perhaps the COM facility is an artifact of an application running as the current user who is logged in, or requires an actual local windows user to run instead.

So by making the tomcat service log on as a local user, I was able to get my COM application to connect from the web application within tomcat.