Detect file handle leaks

[sgane2001]

Hi All,
Is there any way to detect the file handle leaks in java. Though
in TaskMamnager we can see the ope file handles, I want to know the
method causing the leak. Will OptimizeIt be able to find the handle
leaks too.

Searched thro google. But not able to find any useful data.
Thanks,
Ganesh Subramanian

 

[Anton Spaans]

Hi All,
Is there any way to detect the file handle leaks in java. Though
in TaskMamnager we can see the ope file handles, I want to know the
method causing the leak. Will OptimizeIt be able to find the handle
leaks too.

Searched thro google. But not able to find any useful data.
Thanks,
Ganesh Subramanian

I have had no luck using OptimizeIt for using resource leaks. I had a
similar problem and 'Process Explorer' helped me with finding the leak. (Go
to http://www.sysinternals.com/ and download the latest version, it's free).
With Process Explorer, go to the javaw.exe (or java.exe) process that is
your program. You can see the open file handles and their full names (and
other information). I found the leaks based on the names of the open files.

Good luck.
-- Anton.

 

[sgane2001]

Thanks a lot Anton.

But What I'm expecting is a software just similar to Optimize it which
will point me the class in which the handle leak is happening. Since
I'm dealing InputStreams sent by the server as object, Process Explorer
is not showing them.

Is there any other way or a software to find the handle leaks?
Thanks,
Ganesh Subramanian

 

[sgane2001]

Let me make it much clearer. I'm creating icons using ImageIcon which
inturn opens the image file. But I'm not able to see those files in the
'Process Explorer'.

Thanks,
Ganesh Subramanian

 

[Steve W. Jackson]

Let me make it much clearer. I'm creating icons using ImageIcon which
inturn opens the image file. But I'm not able to see those files in the
'Process Explorer'.

Thanks,
Ganesh Subramanian

I've got quite a bit of that kind of code loading image files, and I've
never encountered a problem. So far as I can tell, Java correctly
releases references to those files once the images are loaded.

= Steve =

 

[sgane2001]

But java Caches the image. I'm facing 'too many open files' exception.
So if I open a frame with ToolBar icons, the images won't be loaded. So
the toolbar is shrinkened (set to default size). Even if GC clears the
objects after some time, java will load the image from the cache. So
the toolbar will not be loaded with images. I just want to know where
the leak exactly is. Any help regarding this will be appreciated.
Thanks,
Ganesh Subramanian

 

[sgane2001]

But java Caches the image. I'm facing 'too many open files' exception.
So if I open a frame with ToolBar icons, the images won't be loaded. So
the toolbar is shrinkened (set to default size). Even if GC clears the
objects after some time, java will load the image from the cache. So
the toolbar will not be loaded with images. I just want to know where
the leak exactly is. Any help regarding this will be appreciated.
Thanks,
Ganesh Subramanian

 

[sgane2001]

But java Caches the image. I'm facing 'too many open files' exception.
So if I open a frame with ToolBar icons, the images won't be loaded. So
the toolbar is shrinkened (set to default size). Even if GC clears the
objects after some time, java will load the image from the cache. So
the toolbar will not be loaded with images. I just want to know where
the leak exactly is. Any help regarding this will be appreciated.
Thanks,
Ganesh Subramanian

 

[Anton Spaans]

Let me make it much clearer. I'm creating icons using ImageIcon which
inturn opens the image file. But I'm not able to see those files in the
'Process Explorer'.

Thanks,
Ganesh Subramanian

Then it seems that Java is properly closing the files as soon as the
image-date from those files have been read into the ImageIcon.
While your program is running, monitor the file-handles in Process Explorer
and see what (type of) files are created over and over again, that may be
causing your 'too many files open' error. The issue may *not* be your
ImageIcons.....

(E.g. in our application, it was actually too many named pipes open
(System.err, System.out and System.in for each time an external process
spawned by our app). Closing those processes properly fixed that).

 

[Ann]

Then it seems that Java is properly closing the files as soon as the
image-date from those files have been read into the ImageIcon.
While your program is running, monitor the file-handles in Process Explorer
and see what (type of) files are created over and over again, that may be
causing your 'too many files open' error. The issue may *not* be your
ImageIcons.....

(E.g. in our application, it was actually too many named pipes open
(System.err, System.out and System.in for each time an external process
spawned by our app). Closing those processes properly fixed that).

I don't think the problem is with ImageIcon. I have a program that opens
hundreds of files using
myimage = new ImageIcon(name);
and I don't have this problem.

 

[sgane2001]

I knew that the problem is not because of ImageIcons, But I'm opening a
lot of streams (It's a Mail user Agent. So the worst case can't be
predicted) that are not collected by gc.

(E.g. in our application, it was actually too many named pipes open
(System.err, System.out and System.in for each time an external process
spawned by our app). Closing those processes properly fixed that).

Fine. But if I'm loading a property file like properties.load(new
FileInputStream.....), the Process Explorer is not displaying that
file. Instead an entry with type as thread appears and disappears after
5 secs.

Further, I have faced this problem just twice in a week. I'm not able
to reproduce the problem again. So a software which will show the class
in which the handle leaks will be better.

Or otherwise, could we make the gc to run faster, so it collects the
objects that have no references?

Thanks,
Ganesh Subramanian

 

[Ann]

Or otherwise, could we make the gc to run faster, so it collects the
objects that have no references?

Maybe you can you subclass the garbage collector ;-)

 

[sgane2001]

Unfortunately I'm not in a situation to laugh :-(.
Will running gc in short interval of time work.

 

[John C. Bollinger]

I knew that the problem is not because of ImageIcons, But I'm opening a
lot of streams (It's a Mail user Agent. So the worst case can't be
predicted) that are not collected by gc.

The worst case can be _made_ predictable if you need it to be. For
instance, you could set up a semaphore set to control the number of
concurrent open streams. (Unless you're using Java 5 you'd need to
write your own semaphore implementation or get an external one.)

But if I'm loading a property file like properties.load(new
FileInputStream.....), the Process Explorer is not displaying that
file. Instead an entry with type as thread appears and disappears after
5 secs.

If you need to worry about resource management (i.e. you may be doing
what you describe many times during the lifetime of the application)
then just do it. The pseudocode you offered describes a situation in
which you don't care when or even whether the the stream's resources are
freed. What you probably want is instead something like this:

FileInputStream input = new FileInputStream(fileName);
Properties prop = Properties.load(input);
input.close();

As a general rule, if you instantiate an object that requires cleanup,
then you are responsible for ensuring that it is cleaned up at the
appropriate time. You _may_ be able to fall back on GC to take care of
it for you, but some resources are much more scarce than memory, and GC
may not help you if you are running out of such. Moreover, unless its
class documents so, you cannot expect that _any_ resources other than
memory will be reclaimed when an object is garbage collected.

Further, I have faced this problem just twice in a week. I'm not able
to reproduce the problem again. So a software which will show the class
in which the handle leaks will be better.

How about you just search the source for "InputStream" and
"OutputStream"? This should pick up all the places where you create
such a stream, as well as most, perhaps all, of the places where you
obtain one from another object (such as a Socket or Process) that you
need to take responsibility for.

Or otherwise, could we make the gc to run faster, so it collects the
objects that have no references?

Any Java program that depends on the timing of GC is deeply flawed. Fix
the real problem.


John Bollinger
(e-mail address removed)

 

[Anton Spaans]

I knew that the problem is not because of ImageIcons, But I'm opening a
lot of streams (It's a Mail user Agent. So the worst case can't be
predicted) that are not collected by gc.


Fine. But if I'm loading a property file like properties.load(new
FileInputStream.....), the Process Explorer is not displaying that
file. Instead an entry with type as thread appears and disappears after
5 secs.

Further, I have faced this problem just twice in a week. I'm not able
to reproduce the problem again. So a software which will show the class
in which the handle leaks will be better.

Or otherwise, could we make the gc to run faster, so it collects the
objects that have no references?

Thanks,
Ganesh Subramanian

You have a nasty little problem on your hands.
Maybe the leak only happens when exceptions occur? Maybe the/your code works
fine in 'normal' process flow, but when exceptions occur, streams are not
properly closed (do you use the 'finally {}' clause to close your input- and
output-streams?)? That may be why you see your problem occur only
occasionally.

It sucks that OptimizeIt does not track resources like file-handles! But if
there are 'too many files open', Process Explorer must show a looooooooong
list of file-handles (error comes from operating system)... doesn't it in
your case? If so, which files?

Good luck!

 

[sgane2001]

It sucks that OptimizeIt does not track resources like file-handles!
But if

there are 'too many files open', Process Explorer must show a looooooooong
list of file-handles (error comes from operating system)... doesn't it in
your case? If so, which files?

This is what I got in the Process Explorer. You can't see the open
files. But you can
see a lot of threads running, each of which opens 2-6 handles and have
3 references. The handle count for my process is around 1100.

Process PID CPU Description Company Name Handles
System Idle Process 0 97 0
explorer.exe 1460 Windows Explorer Microsoft Corporation 903
EMMeter.exe 1312 Express Software Manager Express Metrix 230
hkcmd.exe 1608 hkcmd Module Intel Corporation 92
realsched.exe 1560 RealNetworks Scheduler RealNetworks, Inc. 71
ccApp.exe 1620 Common Client User Session Symantec Corporation 284
VPTray.exe 1676 Symantec AntiVirus Symantec Corporation 125
OUTLOOK.EXE 1708 Microsoft Outlook Microsoft Corporation 520
MAPISP32.EXE 1856 Microsoft Windows(TM) Messaging Subsystem
Spooler Microsoft Corporation 195
msimn.exe 1880 Outlook Express Microsoft Corporation 506
aruser.exe 1820 AR System User Application Peregrine Remedy Inc. 139
exec.exe 2256 ZCast NetZero 60
IEXPLORE.EXE 2112 Internet Explorer Microsoft Corporation 608
TASKMGR.EXE 2196 Windows TaskManager Microsoft Corporation 35
procexp.exe 1916 2 Sysinternals Process Explorer Sysinternals 191
java.exe 1636 1,059
DW15.EXE 2044 Microsoft Application Error Reporting Microsoft
Corporation 97

Process: java.exe Pid: 1636

Type Name Handle Access
Desktop \Default 0x3C 0x000F01FF
Directory \KnownDlls 0x14 0x00000003
Directory \BaseNamedObjects 0x160 0x0002000F
Directory \Windows 0x20 0x000F000F
Event \BaseNamedObjects\userenv: User Profile setup
event 0x584 0x001F0003
File C:\Documents and Settings\ksekhar\Local
Settings\Temp\hsperfdata_ksekhar\1636 0x15C 0x0013019F
File C:\ms-mua-7.00\jre\lib\rt.jar 0x180 0x00120089
File C:\ms-mua-7.00\jre\lib\sunrsasign.jar 0x188 0x00120089
File C:\ms-mua-7.00\jre\lib\jsse.jar 0x190 0x00120089
File C:\ms-mua-7.00\jre\lib\jce.jar 0x198 0x00120089
File C:\ms-mua-7.00\jre\lib\charsets.jar 0x1A0 0x00120089
File C:\ms-mua-7.00\jre\lib\ext\dnsns.jar 0x330 0x00120089
File C:\ms-mua-7.00\jre\lib\ext\ldapsec.jar 0x338 0x00120089
File C:\ms-mua-7.00\jre\lib\ext\localedata.jar 0x340 0x00120089
File C:\ms-mua-7.00\jre\lib\ext\sunjce_provider.jar 0x350 0x00120089
File C:\ms-mua-7.00\lib\jhall.jar 0x358 0x00120089
File C:\ms-mua-7.00\lib\BrowserLauncher.jar 0x360 0x00120089
File C:\ms-mua-7.00\lib\mail.jar 0x368 0x00120089
File C:\ms-mua-7.00\lib\ssce.jar 0x370 0x00120089
File C:\ms-mua-7.00\lib\activation.jar 0x378 0x00120089
File C:\ms-mua-7.00\lib\jdom.jar 0x380 0x00120089
File C:\ms-mua-7.00\lib\w3c_http.jar 0x388 0x00120089
File C:\ms-mua-7.00\lib\TableLayout.jar 0x390 0x00120089
File C:\ms-mua-7.00\lib\Application.jar 0x398 0x00120089
File \Device\KsecDD 0x3B0 0x00100001
File C:\WINNT\Fonts\symbol.ttf 0x42C 0x00120089
File C:\ms-mua-7.00\jre\lib\fonts\LucidaSansRegular.ttf 0x430 0x00120089
File C:\WINNT\Fonts\cour.ttf 0x438 0x00120089
File C:\WINNT\Fonts\times.ttf 0x444 0x00120089
File C:\WINNT\Fonts\arial.ttf 0x4F8 0x00120089
File \Device\Tcp 0x500 0x001200A0
File C:\WINNT\Fonts\arialbd.ttf 0x504 0x00120089
File \Device\Tcp 0x5C4 0x001200A0
File \Device\Ip 0x5C8 0x001200A0
File \Device\Tcp 0x5DC 0x001200A0
File \Device\Tcp 0x5FC 0x001200A0
File \Device\Afd 0x600 0x0012019F
File \Device\Tcp 0x604 0x001200A0
File \Device\Afd 0x670 0x0012019F
File C:\WINNT\Fonts\verdana.ttf 0x7F0 0x00120089
File C:\WINNT\Fonts\verdanab.ttf 0x900 0x00120089
File C:\WINNT\Fonts\tahoma.ttf 0x968 0x00120089
File C:\WINNT\Fonts\timesbd.ttf 0x9C4 0x00120089
File \Device\LanmanRedirector\cs1hyd\ksekhar\OUL
SWs\ju-cw2.exe 0xAB4 0x00120089
File \Dfs 0xB9C 0x00100000
File \Device\LanmanRedirector\cs1hyd\ksekhar\OUL
SWs\JunoSetup.exe 0xD14 0x00120089
File \Device\LanmanRedirector\cs1hyd\ksekhar\OUL
SWs\JunoSetup.exe 0xE20 0x00120089
File \Device\LanmanRedirector\cs1hyd\ksekhar\OUL
SWs\ju-cw2.exe 0xE94 0x00120089
File C:\Program Files\Common Files\System\Mapi\1033\NT 0xF6C 0x00100020
Key HKLM 0x28 0x00020019
Key HKCU 0x2E4 0x000F003F
Key HKLM\SYSTEM\ControlSet001\Control\NetworkProvider\HwOrder 0x3A4 0x00020019
Key HKLM\SOFTWARE\MICROSOFT\WINDOWS
NT\CURRENTVERSION\DRIVERS32 0x48 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\WinSock2\Parameters\Protocol_Catalog9 0x4B8 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\WinSock2\Parameters\NameSpace_Catalog5 0x574 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\Tcpip\Linkage 0x5CC 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\Tcpip\Parameters 0x5D0 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\NetBT\Parameters\Interfaces 0x5D4 0x00020019
Key HKLM\SYSTEM\ControlSet001\Services\NetBT\Parameters 0x5D8 0x00020019
Key HKCU\Software\Classes 0xACC 0x000F003F
Key HKLM\SOFTWARE\MICROSOFT\Windows\CURRENTVERSION\Explorer 0xAD4 0x00020019
Key HKCU\Software\Classes 0xAD8 0x000F003F
Key HKCR 0xADC 0x00020019
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xAE4 0x00020019
Key HKU 0xAEC 0x00000010
Key HKCR 0xAF4 0x00020019
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xAFC 0x00020019
Key HKU 0xB04 0x00000010
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xB0C 0x00020019
Key HKCR\CLSID 0xB14 0x00020019
Key HKCR 0xB1C 0x00020019
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xB24 0x00020019
Key HKU 0xB2C 0x00000010
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xB34 0x00020019
Key HKLM\SOFTWARE\MICROSOFT\COM3 0xB3C 0x00020019
Key HKCR\CLSID 0xB44 0x00020019
Key HKCU\Software\Classes 0xB54 0x00020019
Key HKCU\SOFTWARE\MICROSOFT\Windows\ShellNoRoam\MUICache 0xB8C 0x000F003F
Key HKCU\SOFTWARE\MICROSOFT\Windows\ShellNoRoam 0xB90 0x000F003F
Key HKCU\Software\Classes 0xBB8 0x00020019
Key HKCU\Software\Classes 0xE2C 0x00020019
Mutant \BaseNamedObjects\DDrawWindowListMutex 0x3B4 0x001F0001
Mutant \BaseNamedObjects\DDrawDriverObjectListMutex 0x448 0x001F0001
Mutant \BaseNamedObjects\__DDrawExclMode__ 0x44C 0x001F0001
Mutant \BaseNamedObjects\__DDrawCheckExclMode__ 0x450 0x001F0001
Mutant \BaseNamedObjects\RasPbFile 0x5A8 0x00100000
Port \RPC Control\OLEAB8D281614AC42739CCF85D779BE 0xB60 0x001F0001
Section \BaseNamedObjects\hsperfdata_ksekhar_1636 0x164 0x000F0007
Section \BaseNamedObjects\__R_0000000000cc_SMem__ 0xB4C 0x00000004
Semaphore \BaseNamedObjects\shell.{210A4BA0-3AEA-1069-A2D9-08002B30309D} 0xAC4 0x00100002
Semaphore \BaseNamedObjects\shell.{A48F1A32-A340-11D1-BC6B-00A0C90312E1} 0xAC8 0x00100002
Semaphore \BaseNamedObjects\shell.{7CB834F0-527B-11D2-9D1F-0000F805CA57} 0xAD0 0x00100002
Semaphore \BaseNamedObjects\shell.{090851A5-EB96-11D2-8BE4-00C04FA31A66} 0xB80 0x00100002
Thread java.exe(1636): 2292 0x1000 0x001F03FF
Thread java.exe(1636): 2208 0x1004 0x001F03FF
Thread java.exe(1636): 2276 0x1008 0x001F03FF
Thread java.exe(1636): 2096 0x1010 0x001F03FF
Thread java.exe(1636): 1952 0x1030 0x001F03FF
Thread java.exe(1636): 656 0x1038 0x001F03FF
Thread java.exe(1636): 2328 0x103C 0x001F03FF
Thread java.exe(1636): 1992 0x1058 0x001F03FF
Thread java.exe(1636): 2008 0x1068 0x001F03FF
Thread java.exe(1636): 2468 0x107C 0x001F03FF
Thread java.exe(1636): 2140 0x108C 0x001F03FF
Thread java.exe(1636): 1632 0x174 0x001F03FF
Thread java.exe(1636): 1740 0x1C0 0x001F03FF
Thread java.exe(1636): 1812 0x1D0 0x001F03FF
Thread java.exe(1636): 1816 0x2E0 0x001F03FF
Thread java.exe(1636): 1836 0x2F4 0x001F03FF
Thread java.exe(1636): 1840 0x308 0x001F03FF
Thread java.exe(1636): 1844 0x318 0x001F03FF
Thread java.exe(1636): 1848 0x328 0x001F03FF
Thread java.exe(1636): 1524 0x478 0x001F03FF
Thread java.exe(1636): 1552 0x4CC 0x001F03FF
Thread java.exe(1636): 1800 0x4E8 0x001F03FF
Thread java.exe(1636): 1872 0x510 0x001F03FF
Thread java.exe(1636): 1632 0x668 0x001F03FF
Thread java.exe(1636): 1940 0x680 0x001F03FF
Thread java.exe(1636): 1940 0x684 0x001F03FF
Thread java.exe(1636): 1860 0xA78 0x001F03FF
Thread java.exe(1636): 1452 0xB70 0x001F03FF
Thread java.exe(1636): 1556 0xBD4 0x001F03FF
Thread java.exe(1636): 1136 0xC40 0x001F03FF
Thread java.exe(1636): 1804 0xC80 0x001F03FF
Thread java.exe(1636): 1616 0xCA0 0x001F03FF
Thread java.exe(1636): 1684 0xD04 0x001F03FF
Thread java.exe(1636): 1588 0xD28 0x001F03FF
Thread java.exe(1636): 1572 0xD38 0x001F03FF
Thread java.exe(1636): 1788 0xD74 0x001F03FF
Thread java.exe(1636): 492 0xD94 0x001F03FF
Thread java.exe(1636): 860 0xDA8 0x001F03FF
Thread java.exe(1636): 1340 0xDC0 0x001F03FF
Thread java.exe(1636): 988 0xDDC 0x001F03FF
Thread java.exe(1636): 804 0xE04 0x001F03FF
Thread java.exe(1636): 296 0xE0C 0x001F03FF
Thread java.exe(1636): 1948 0xE40 0x001F03FF
Thread java.exe(1636): 2068 0xE54 0x001F03FF
Thread java.exe(1636): 1988 0xE68 0x001F03FF
Thread java.exe(1636): 1492 0xE74 0x001F03FF
Thread java.exe(1636): 2124 0xE88 0x001F03FF
Thread java.exe(1636): 2152 0xEB0 0x001F03FF
Thread java.exe(1636): 2156 0xEB4 0x001F03FF
Thread java.exe(1636): 2284 0xEF8 0x001F03FF
Thread java.exe(1636): 2252 0xF00 0x001F03FF
Thread java.exe(1636): 1348 0xF18 0x001F03FF
Thread java.exe(1636): 2180 0xF2C 0x001F03FF
Thread java.exe(1636): 2280 0xF44 0x001F03FF
Thread java.exe(1636): 644 0xF4C 0x001F03FF
Thread java.exe(1636): 644 0xF50 0x001F03FF
Thread java.exe(1636): 644 0xF64 0x001F03FF
Thread java.exe(1636): 2136 0xFA8 0x001F03FF
Thread java.exe(1636): 1784 0xFB4 0x001F03FF
Thread java.exe(1636): 1476 0xFC4 0x001F03FF
Thread java.exe(1636): 896 0xFD8 0x001F03FF
WindowStation \Windows\WindowStations\WinSta0 0x38 0x000F037F
WindowStation \Windows\WindowStations\WinSta0 0x40 0x000F037F

How about you just search the source for "InputStream" and
"OutputStream"? This should pick up all the places where you create
such a stream, as well as most, perhaps all, of the places where you
obtain one from another object (such as a Socket or Process) that you
need to take responsibility for.

I tried the same. But I'm not able to figure out the area. I'm using
finally{} at all the places. I thinks some one is running as thread
that is waiting for some other to exit.

Thanks,
Ganesh Subramanian

 

[sgane2001]

Finally fixed the problem. This was due to a reference to
Mimemessage.getInputstream() which inturn calls
Datahandler.inputStream(). Datahandler opens a thread and a pipe. The
thread was waiting for the inputstream to get collected. So for each
and every reply, the handle count increased by 2.

My Thanks to all of you and OptimizeIt

Ganesh Subramanian

 

[pavel.genevski]

Hi guys,

at our team, we have encountered very similar problems with leaking file handles in the past, so I decided to create a tool for I/O analysis that, amongst other things, helps find out which threads have opened the handle, at what time they did it etc.

The tool is named JPicus. It is currently available for download, free of charge, at the SAP Developer Network Wiki. Just search for JPicus and you shall be able to find it. I would be glad if you try it and share your experience.

Regards,
Pavel Genevski