thread release memory

[Matt Goodwin]

I have a program that looks at a directory for files and if there are
some files it reads the file contents into a byte stream and calls a
web service method. The problem I am having is that system memory
never gets released. I have profiled the application with JProbe and
memory gets collected from the heap, but when I look at the Windows
Task Manager it keeps going up and up until I get an out of memory
exception. The weird thing is when I comment out the web service call
the memory never climbs. I think I am getting rid of all references
to objects. Below are some relevant pieces of the code.

// code snippet from main method

Thread t1 = new Thread() {
public void run() {
while(true) {
try {
List files = DirectoryScanner.getFiles(dirName,fileExt,startTime); //
static method
Iterator itr = files.iterator();
while(itr.hasNext()) {
final File file = (File)itr.next();
log.info("Filename: " + file.getName());
new Thread(new Runnable() {
public synchronized void run() {
FileTransfer.TransferFile(file, logDirName, failedDirName,
successDirName); //static method
}}).start();
}
files=null;
sleep(interval);
} catch(Exception e) {
log.info("Caught exception setting up receiver " + e.getMessage());
e.printStackTrace();
}
}
}
};
t1.setName("DirectoryScanner Thread");
t1.start();

// code from FileTransfer.TransferFile method
public static void TransferFile(File file, String logDirName, String
failedDirName, String successDirName ) {
String fileHeader=null;
String fileContents=null;
TransformFileSoap binding=null;
// read file
try {
fileHeader = getFileHeader(file.getName());
if(fileHeader==null){
errLog.error("Filename: " + file.getAbsolutePath() + " not using
correct naming convention");
log.error("Filename: " + file.getAbsolutePath() + " not using correct
naming convention");
return;
}
fileContents = getFileContents(file);
if(fileContents==null){
errLog.error("Could not retrieve contents of file: " +
file.getAbsolutePath());
log.error("Could not retrieve contents of file: " +
file.getAbsolutePath());
return;
}
// call web service
try {
binding = new TransformFileLocator().getTransformFileSoap();
((TransformFileSoapStub)binding).setMaintainSession(false);
} catch(javax.xml.rpc.ServiceException se) {
errLog.error("ServiceException: Could not locate service: " +
se.getMessage());
moveToDirectory(file,failedDirName);
return;
}
try {
binding.convert(fileContents,fileHeader);
} catch(java.rmi.RemoteException jre) {
log.error(jre.getMessage());
errLog.error("File: " + file.getName() + " " + jre.getMessage());
moveToDirectory(file,failedDirName);
return;
}
moveToDirectory(file,successDirName);
appLog.info("File: " + file.getName() + " successfully processed");
} catch(Exception e) {
log.error("Could not process file: " + file.getName() +
e.getMessage());
errLog.error("Could not process file: " + file.getName() +
e.getMessage());
} finally {
fileHeader=null;
fileContents=null;
binding=null;
}
}

Thanks in advance,
Matt

 

[John C. Bollinger]

I have a program that looks at a directory for files and if there are
some files it reads the file contents into a byte stream and calls a
web service method. The problem I am having is that system memory
never gets released. I have profiled the application with JProbe and
memory gets collected from the heap, but when I look at the Windows
Task Manager it keeps going up and up until I get an out of memory
exception.

It is possible -- likely, even -- that memory freed by GC does not get
returned to the system until the VM exits, but if it is freed then it is
by definition available to the VM for the heap, and indeed, is probably
already _in_ the heap. If you get an OutOfMemoryError then it indicates
that you have insufficient free space in the heap, which suggests that
you either are trying to create some extremely large objects or are
hanging on to a lot of smaller objects that are _not_ eligible for GC.

The weird thing is when I comment out the web service call
the memory never climbs. I think I am getting rid of all references
to objects.

You probably think wrongly, then. Chances are that references are being
retained that are invisible to you. For instance, if you create a
substring of a larger string then the two String objects share a
reference to the backing char[]. If the original string is large then
even after discarding all references to it, the majority of its memory
may still be tied up by the substring. If you intern() Strings then you
permanently tie up memory; unless your code is carefully designed and
written this is probably pure waste.

Below are some relevant pieces of the code.
[...]

// code from FileTransfer.TransferFile method
public static void TransferFile(File file, String logDirName, String
failedDirName, String successDirName ) {
String fileHeader=null;
String fileContents=null;
TransformFileSoap binding=null;
// read file
try {
fileHeader = getFileHeader(file.getName());
if(fileHeader==null){
errLog.error("Filename: " + file.getAbsolutePath() + " not using
correct naming convention");
log.error("Filename: " + file.getAbsolutePath() + " not using correct
naming convention");
return;
}
fileContents = getFileContents(file);

Reading whole files into Strings is not good form, and may be related to
your problem. (Remember what I wrote about substrings?) It is almost
always better to handle a file as a stream of bytes / chars (whichever
is appropriate).

if(fileContents==null){
errLog.error("Could not retrieve contents of file: " +
file.getAbsolutePath());
log.error("Could not retrieve contents of file: " +
file.getAbsolutePath());
return;
}
// call web service
try {
binding = new TransformFileLocator().getTransformFileSoap();
((TransformFileSoapStub)binding).setMaintainSession(false);
} catch(javax.xml.rpc.ServiceException se) {
errLog.error("ServiceException: Could not locate service: " +
se.getMessage());
moveToDirectory(file,failedDirName);
return;
}
try {
binding.convert(fileContents,fileHeader);

And do I understand correctly that commenting out just the one line
above causes the program to not manifest the problem? It would seem,
then, that you have neglected to provide the most important parts of the
problematic code.

} catch(java.rmi.RemoteException jre) {
log.error(jre.getMessage());
errLog.error("File: " + file.getName() + " " + jre.getMessage());
moveToDirectory(file,failedDirName);
return;
}
moveToDirectory(file,successDirName);
appLog.info("File: " + file.getName() + " successfully processed");
} catch(Exception e) {
log.error("Could not process file: " + file.getName() +
e.getMessage());
errLog.error("Could not process file: " + file.getName() +
e.getMessage());
} finally {
fileHeader=null;
fileContents=null;
binding=null;

This explicit nulling is unnecessary; your method is about to return
anyway, after which all local variables associated with the current
invocation will effectively cease to exist.

}
}

John Bollinger
(e-mail address removed)

 

[Roedy Green]

hances are that references are being
retained that are invisible to you.

see http://mindprod.com/jgloss/packratting.html

 

[Matt Goodwin]

You probably think wrongly, then. Chances are that references are being
retained that are invisible to you. For instance, if you create a
substring of a larger string then the two String objects share a
reference to the backing char[]. If the original string is large then
even after discarding all references to it, the majority of its memory
may still be tied up by the substring. If you intern() Strings then you
permanently tie up memory; unless your code is carefully designed and
written this is probably pure waste.

good call. i was doing some substringing of a file name. thanks.

Reading whole files into Strings is not good form, and may be related to
your problem. (Remember what I wrote about substrings?) It is almost
always better to handle a file as a stream of bytes / chars (whichever
is appropriate).

i am reading the file into a byte array and then creating a new string
because that's what the web service (i didn't write the web service)
required. when i took out converting to a string the memory
consumption went way down from 140m on the heap to next to nothing
(the size of the byte array). i'm talking to the author of the web
service about having the service take a byte array instead. thanks
again.

And do I understand correctly that commenting out just the one line
above causes the program to not manifest the problem? It would seem,
then, that you have neglected to provide the most important parts of the
problematic code.

sorry i didn't post the code because it was generated by wsdl2java and
after inspecting it i didn't see how it could be retaining references.
here it is:
public String convert(java.lang.String daFile, java.lang.String
daFileType)
throws java.rmi.RemoteException {
if (super.cachedEndpoint == null) {
throw new org.apache.axis.NoEndPointException();
}
org.apache.axis.client.Call _call = createCall();
_call.setOperation(_operations[0]);
_call.setUseSOAPAction(true);
_call.setSOAPActionURI("http://biztalk.equitrust.com/transformfile/Convert");
_call.setEncodingStyle(null);
_call.setProperty(org.apache.axis.client.Call.SEND_TYPE_ATTR,
Boolean.FALSE);
_call.setProperty(org.apache.axis.AxisEngine.PROP_DOMULTIREFS,
Boolean.FALSE);
_call.setSOAPVersion(org.apache.axis.soap.SOAPConstants.SOAP11_CONSTANTS);
_call.setOperationName(new
javax.xml.namespace.QName("http://biztalk.equitrust.com/transformfile/",
"Convert"));

setRequestHeaders(_call);
setAttachments(_call);
java.lang.Object _resp = _call.invoke(new java.lang.Object[]
{daFile, daFileType});

if (_resp instanceof java.rmi.RemoteException) {
throw (java.rmi.RemoteException)_resp;
}
else {
extractAttachments(_call);
try {
return (java.lang.String) _resp;
} catch (java.lang.Exception _exception) {
return (java.lang.String)
org.apache.axis.utils.JavaUtils.convert(_resp,
java.lang.String.class);
}
}
}

This explicit nulling is unnecessary; your method is about to return
anyway, after which all local variables associated with the current
invocation will effectively cease to exist.

understood.

thanks.
matt