URLConnection "memory leak"

T

timjowers

URLConnection leaks memory and after a while will return the memory. No
amount of calling System.gc(); will clean it up. OK for small apps but
I am writing one that downloads 100 or so pages synchronously. OK
unless I experience several clients asking my server to do this at
once. Then Java gives the almost useless OutOfMemoryError. So much for
Garbage Collection. At this point GC just means random
non-deterministic memory management and means, unlike in C++, I cannot
achieve a robust server application. How can Java justify giving
OutOfMemoryError when NO REFERENCES ARE HELD IN MY PROGRAM!!!! I guess
URLConnection must have some internal static buggy code.

Any ideas how to get URLConnection to stop hanging onto memory?

Here's some test code below.

TimJowers

/**
* Sample memory
*/
import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;



public class Leaky {

public String strXML = null;
private String proxyHost=null, proxyPort=null, username=null,
password=null,
httpRequest=null, base64Encoded=null, proxyType="2",
formValues=null, formMethod="", optionFile=null;
private URL url=null;
private URLConnection urlConnection=null;
private InputStream is=null;
private File file=null;
private FileOutputStream out=null;
private int ftpPort=21;
private boolean notAssigned = true,
ftpTrue=false, ftpPassive=false, ftpAscii=false,
https=false, nonverbose=true, post=false, booleanOptionFile=false;
private StringBuffer sbuf = null;
private byte[] bytesRead = new byte[4096];

protected void zeroMem()
{
strXML = null;
proxyHost=null; proxyPort=null; username=null; password=null;
httpRequest=null; base64Encoded=null;
formValues=null;
formMethod=""; optionFile=null;
url=null;
urlConnection=null;
is=null;
file=null;
out=null;
sbuf = null;
bytesRead = null;
}

public URLConnection setupProxy ( String [] args ) throws Exception
{
getOptions(args);

if (https) {
if (!nonverbose) {
System.out.println("Using https communication");
}
Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
System.setProperty("http.keepAlive", "false");
System.setProperty( "java.protocol.handler.pkgs",
"com.sun.net.ssl.internal.www.protocol" );
}
if (formMethod != null && formMethod.equalsIgnoreCase("get")) {
httpRequest = httpRequest + "?" + encodeHttpString(formValues);
}
try {
url = new URL(httpRequest);
} catch (MalformedURLException e) {
if (!nonverbose) {
System.err.println("Malformed URL, must be
http://somehost[/file]");
} else {
throw new Exception("Malformed URL, must be
http://somehost[/file]");
}
return null;
}
if (proxyHost != null) {
if (proxyType.equals("1") || proxyType.equals("3")) {
System.getProperties().put( "proxySet", "true" );
System.getProperties().put( "proxyHost", proxyHost);
System.getProperties().put( "proxyPort", proxyPort);
} else {
if (proxyType.equals("4")) {
System.getProperties().put( "socksProxySet", "true");
System.getProperties().put( "socksProxyHost", proxyHost);
System.getProperties().put( "SocksProxyPort" , proxyPort);
} else {
if (proxyType.equals("5")) {
System.getProperties().put( "ftpProxySet", "true" );
System.getProperties().put( "ftpProxyHost", proxyHost );
System.getProperties().put( "ftpProxyPort", proxyPort );
} else {
System.getProperties().put("firewallSet", "true");
System.getProperties().put("firewallHost", proxyHost);
System.getProperties().put("firewallPort", proxyPort);
System.getProperties().put("http.proxyHost", proxyHost);
System.getProperties().put("http.proxyPort", proxyPort);
}}}
if (https && proxyHost != null) {
System.getProperties().put("https.proxyHost", proxyHost);
System.getProperties().put("https.proxyPort", proxyPort);
}
}
try {
urlConnection = url.openConnection();
} catch (IOException e) {
if (!nonverbose) {
System.err.println("Error opening URL connection to " +
httpRequest);
} else {
throw new Exception("Error opening URL Connection to " +
httpRequest);
}
return null;
}
if (proxyHost != null) {
Base64 base64 = new Base64(username + ":" + password);
base64.encode();
base64Encoded = "Basic " + base64.getOutgoing();
urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
if (proxyType.equals("3")) {
urlConnection.setRequestProperty("Authorization",
base64Encoded);
} else {
urlConnection.setRequestProperty("Proxy-Authorization",
base64Encoded);
}
}
return urlConnection;
}

public boolean doWget ( String args ) throws Exception {
// if args are all on one string then break into an array of strings
StringTokenizer strTok = new StringTokenizer( args );
String []argv = new String[ strTok.countTokens() ];
for(int i=0; strTok.hasMoreTokens(); i++ )
argv = strTok.nextToken();
return doWget( argv );
}

public boolean doWget ( String [] args ) throws Exception {

System.out.println(httpRequest);
System.out.println(proxyHost);
System.out.println(ftpPort);

urlConnection = setupProxy ( args );

if (!nonverbose) {
System.out.println("Connecting to " + httpRequest + "...");
if(
0==httpRequest.compareTo("http://www.p2p.wrox.com/listindex.asp") ) //
hangs
return false;
if(
0==httpRequest.compareTo("http://www.soton.ac.uk/~chst/direct.htm") )
// hangs
return false;
if(
0==httpRequest.compareTo("https://www.ahamembership.com/assoc6/k.cgi?p=10-TX&acct_code=TX004-10TX-T")
) // security exception
return false;
if(
0==httpRequest.compareTo("https://www25.hway.net/onl261/cgi-local/sgx/shop.cgi?page=order.html&afnum=21")
) // security exception
return false;
if( 0==httpRequest.compareTo("https://grc.com/x/ne.dll?bh0bkyd2") )
// security exception
return false;
}
try
{
DataOutputStream printout;
DataInputStream input;

urlConnection.setDoInput (true);
urlConnection.setUseCaches (false);
urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
urlConnection.setRequestProperty("Cache-Control","no-cache");

if (null != strXML) // add XML data as HTTP payload
{ System.out.println("Adding XML...");
// Send POST output.
printout = new DataOutputStream (urlConnection.getOutputStream ());
String content =
"CoNUM=" + URLEncoder.encode ("2") +
"&PASSWORD=" + URLEncoder.encode ("1");
String msg;
msg = "<?xml version=" + "\"" + "1.0" + "\"" + "?>";
msg = msg + "</XML>";
content = msg;
System.out.println( " --- XML to Web Server ---\r\r" + content );
printout.writeBytes (content);
printout.flush ();
printout.close ();
}
}
catch (MalformedURLException me)
{
System.err.println("MalformedURLException: " + me);
}
catch (IOException ioe)
{
System.err.println("IOException: " + ioe.getMessage());
}
if( !readHttp() )
return false;
closeAll();
return true;
}

private void getOptions( String[] args ) throws Exception {
for (int i=0; i<args.length; i++) {
if (args.equals("-h")) {
StringTokenizer st = new StringTokenizer(args[i+1], ":");
int count=0;
while (st.hasMoreTokens()) {
if (count == 0 ) {
proxyHost=st.nextToken();
}
if (count == 1) {
proxyPort=st.nextToken();
}
count++;
}
if (proxyPort == null) {
proxyPort = "80";
}
i++;
notAssigned=false;
}
if (args.equals("-u")) {
username=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-p")) {
password=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-t")) {
proxyType=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-post")) {
formValues=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-method")) {
formMethod=args[i+1];
i++;
notAssigned=false;
}
if( args.equals("-xml") ) // get XML file
{ strXML = new String("<XML></XML>");
}
if (proxyHost != null && ftpTrue==true) {
if (!nonverbose) {
System.err.println("Proxy and Ftp cannot coexist use -h or -ftp");
} else {
throw new Exception("Proxy and Ftp cannot coexist use -h or
-ftp");
}
return;
}
if (notAssigned) {
httpRequest=args;
if (httpRequest.substring(0, 5).equalsIgnoreCase("https")) {
https=true;
}
}
notAssigned=true;
}
}

/** Main invoked from the prompt.
* @param args Arguments from the Stdin
* @throws Exception Throws Exception
*/
static String[] urls = {"http://www.youart.net/",
"http://www.microsoft.net/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.cnn.com/"};
public static void main ( String[] args ) throws Exception {

for( int e=0; e<urls.length; e++)
{ Leaky wget = new Leaky();
wget.doWget( urls[e] );
String result = wget.getOutput();
System.out.println( "page has " + result.length() + " characters" );
wget.zeroMem();
wget=null;
System.gc();
System.gc();
System.gc();
}
// memory is leaked at this point. Next memory is magically returned
if we sleep awhile.
for( int s=0; s<60; s++ ) {
try{ Thread.sleep(60000);}catch(Exception e){}
System.gc();
}
}
 
A

Andrew Thompson

On 2 Jun 2005 21:51:37 -0700, (e-mail address removed) wrote:

....
public String strXML = null;
Click to expand...

Please refrain form posting tab characters to usenet, as they are
often expanded to ridiculous lengths by news reader software. And
keep your line lengths short. Otherwise lines wrap, and therefore break..
....
System.err.println("Malformed URL, must be
http://somehost[/file]");
Click to expand...

Also, you are missing a closing '}'. If you had stuck to a single
convention for formatting code, this might be easier to spot, but it
seems you are mixing and matching..

When those problems are fixed, I get..

C:\..\Leaky.java:60: cannot find symbol
symbol : method encodeHttpString(java.lang.String)
location: class Leaky
httpRequest = httpRequest + "?" +
encodeHttpString(formValues);
^
C:\..\Leaky.java:114: cannot find symbol
symbol : class Base64
location: class Leaky
Base64 base64 = new Base64(username + ":" +
password);
^
C:\..\Leaky.java:114: cannot find symbol
symbol : class Base64
location: class Leaky
Base64 base64 = new Base64(username + ":" +
password);
^
C:\..\Leaky.java:203: cannot find symbol
symbol : method readHttp()
location: class Leaky
if( !readHttp() )
^
C:\..\Leaky.java:205: cannot find symbol
symbol : method closeAll()
location: class Leaky
closeAll();
^
C:\..\Leaky.java:321: cannot find symbol
symbol : method getOutput()
location: class Leaky
String result = wget.getOutput();
^
Note: C:\..\Leaky.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
6 errors

....It seems you might benefit from a document specifically prepared
to help people prepare a good example of a problem. I suggest
(and recommend*) this one. <http://www.physci.org/codes/sscce.jsp>

* Not surprising, since I wrote and host it.

As far as the actual problem, I have no insights. I was curious
to see your code break, but do not have a great deal of experience
with URLConnection (at least, not in terms of memory leaks).
 
K

Knute Johnson

URLConnection leaks memory and after a while will return the memory. No
amount of calling System.gc(); will clean it up. OK for small apps but
I am writing one that downloads 100 or so pages synchronously. OK
unless I experience several clients asking my server to do this at
once. Then Java gives the almost useless OutOfMemoryError. So much for
Garbage Collection. At this point GC just means random
non-deterministic memory management and means, unlike in C++, I cannot
achieve a robust server application. How can Java justify giving
OutOfMemoryError when NO REFERENCES ARE HELD IN MY PROGRAM!!!! I guess
URLConnection must have some internal static buggy code.

Any ideas how to get URLConnection to stop hanging onto memory?

Here's some test code below.

TimJowers

/**
* Sample memory
*/
import java.net.*;
import java.io.*;
import java.util.*;
import java.security.*;



public class Leaky {

public String strXML = null;
private String proxyHost=null, proxyPort=null, username=null,
password=null,
httpRequest=null, base64Encoded=null, proxyType="2",
formValues=null, formMethod="", optionFile=null;
private URL url=null;
private URLConnection urlConnection=null;
private InputStream is=null;
private File file=null;
private FileOutputStream out=null;
private int ftpPort=21;
private boolean notAssigned = true,
ftpTrue=false, ftpPassive=false, ftpAscii=false,
https=false, nonverbose=true, post=false, booleanOptionFile=false;
private StringBuffer sbuf = null;
private byte[] bytesRead = new byte[4096];

protected void zeroMem()
{
strXML = null;
proxyHost=null; proxyPort=null; username=null; password=null;
httpRequest=null; base64Encoded=null;
formValues=null;
formMethod=""; optionFile=null;
url=null;
urlConnection=null;
is=null;
file=null;
out=null;
sbuf = null;
bytesRead = null;
}

public URLConnection setupProxy ( String [] args ) throws Exception
{
getOptions(args);

if (https) {
if (!nonverbose) {
System.out.println("Using https communication");
}
Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
System.setProperty("http.keepAlive", "false");
System.setProperty( "java.protocol.handler.pkgs",
"com.sun.net.ssl.internal.www.protocol" );
}
if (formMethod != null && formMethod.equalsIgnoreCase("get")) {
httpRequest = httpRequest + "?" + encodeHttpString(formValues);
}
try {
url = new URL(httpRequest);
} catch (MalformedURLException e) {
if (!nonverbose) {
System.err.println("Malformed URL, must be
http://somehost[/file]");
} else {
throw new Exception("Malformed URL, must be
http://somehost[/file]");
}
return null;
}
if (proxyHost != null) {
if (proxyType.equals("1") || proxyType.equals("3")) {
System.getProperties().put( "proxySet", "true" );
System.getProperties().put( "proxyHost", proxyHost);
System.getProperties().put( "proxyPort", proxyPort);
} else {
if (proxyType.equals("4")) {
System.getProperties().put( "socksProxySet", "true");
System.getProperties().put( "socksProxyHost", proxyHost);
System.getProperties().put( "SocksProxyPort" , proxyPort);
} else {
if (proxyType.equals("5")) {
System.getProperties().put( "ftpProxySet", "true" );
System.getProperties().put( "ftpProxyHost", proxyHost );
System.getProperties().put( "ftpProxyPort", proxyPort );
} else {
System.getProperties().put("firewallSet", "true");
System.getProperties().put("firewallHost", proxyHost);
System.getProperties().put("firewallPort", proxyPort);
System.getProperties().put("http.proxyHost", proxyHost);
System.getProperties().put("http.proxyPort", proxyPort);
}}}
if (https && proxyHost != null) {
System.getProperties().put("https.proxyHost", proxyHost);
System.getProperties().put("https.proxyPort", proxyPort);
}
}
try {
urlConnection = url.openConnection();
} catch (IOException e) {
if (!nonverbose) {
System.err.println("Error opening URL connection to " +
httpRequest);
} else {
throw new Exception("Error opening URL Connection to " +
httpRequest);
}
return null;
}
if (proxyHost != null) {
Base64 base64 = new Base64(username + ":" + password);
base64.encode();
base64Encoded = "Basic " + base64.getOutgoing();
urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
if (proxyType.equals("3")) {
urlConnection.setRequestProperty("Authorization",
base64Encoded);
} else {
urlConnection.setRequestProperty("Proxy-Authorization",
base64Encoded);
}
}
return urlConnection;
}

public boolean doWget ( String args ) throws Exception {
// if args are all on one string then break into an array of strings
StringTokenizer strTok = new StringTokenizer( args );
String []argv = new String[ strTok.countTokens() ];
for(int i=0; strTok.hasMoreTokens(); i++ )
argv = strTok.nextToken();
return doWget( argv );
}

public boolean doWget ( String [] args ) throws Exception {

System.out.println(httpRequest);
System.out.println(proxyHost);
System.out.println(ftpPort);

urlConnection = setupProxy ( args );

if (!nonverbose) {
System.out.println("Connecting to " + httpRequest + "...");
if(
0==httpRequest.compareTo("http://www.p2p.wrox.com/listindex.asp") ) //
hangs
return false;
if(
0==httpRequest.compareTo("http://www.soton.ac.uk/~chst/direct.htm") )
// hangs
return false;
if(
0==httpRequest.compareTo("https://www.ahamembership.com/assoc6/k.cgi?p=10-TX&acct_code=TX004-10TX-T")
) // security exception
return false;
if(
0==httpRequest.compareTo("https://www25.hway.net/onl261/cgi-local/sgx/shop.cgi?page=order.html&afnum=21")
) // security exception
return false;
if( 0==httpRequest.compareTo("https://grc.com/x/ne.dll?bh0bkyd2") )
// security exception
return false;
}
try
{
DataOutputStream printout;
DataInputStream input;

urlConnection.setDoInput (true);
urlConnection.setUseCaches (false);
urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
urlConnection.setRequestProperty("Cache-Control","no-cache");

if (null != strXML) // add XML data as HTTP payload
{ System.out.println("Adding XML...");
// Send POST output.
printout = new DataOutputStream (urlConnection.getOutputStream ());
String content =
"CoNUM=" + URLEncoder.encode ("2") +
"&PASSWORD=" + URLEncoder.encode ("1");
String msg;
msg = "<?xml version=" + "\"" + "1.0" + "\"" + "?>";
msg = msg + "</XML>";
content = msg;
System.out.println( " --- XML to Web Server ---\r\r" + content );
printout.writeBytes (content);
printout.flush ();
printout.close ();
}
}
catch (MalformedURLException me)
{
System.err.println("MalformedURLException: " + me);
}
catch (IOException ioe)
{
System.err.println("IOException: " + ioe.getMessage());
}
if( !readHttp() )
return false;
closeAll();
return true;
}

private void getOptions( String[] args ) throws Exception {
for (int i=0; i<args.length; i++) {
if (args.equals("-h")) {
StringTokenizer st = new StringTokenizer(args[i+1], ":");
int count=0;
while (st.hasMoreTokens()) {
if (count == 0 ) {
proxyHost=st.nextToken();
}
if (count == 1) {
proxyPort=st.nextToken();
}
count++;
}
if (proxyPort == null) {
proxyPort = "80";
}
i++;
notAssigned=false;
}
if (args.equals("-u")) {
username=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-p")) {
password=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-t")) {
proxyType=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-post")) {
formValues=args[i+1];
i++;
notAssigned=false;
}
if (args.equals("-method")) {
formMethod=args[i+1];
i++;
notAssigned=false;
}
if( args.equals("-xml") ) // get XML file
{ strXML = new String("<XML></XML>");
}
if (proxyHost != null && ftpTrue==true) {
if (!nonverbose) {
System.err.println("Proxy and Ftp cannot coexist use -h or -ftp");
} else {
throw new Exception("Proxy and Ftp cannot coexist use -h or
-ftp");
}
return;
}
if (notAssigned) {
httpRequest=args;
if (httpRequest.substring(0, 5).equalsIgnoreCase("https")) {
https=true;
}
}
notAssigned=true;
}
}

/** Main invoked from the prompt.
* @param args Arguments from the Stdin
* @throws Exception Throws Exception
*/
static String[] urls = {"http://www.youart.net/",
"http://www.microsoft.net/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.cnn.com/"};
public static void main ( String[] args ) throws Exception {

for( int e=0; e<urls.length; e++)
{ Leaky wget = new Leaky();
wget.doWget( urls[e] );
String result = wget.getOutput();
System.out.println( "page has " + result.length() + " characters" );
wget.zeroMem();
wget=null;
System.gc();
System.gc();
System.gc();
}
// memory is leaked at this point. Next memory is magically returned
if we sleep awhile.
for( int s=0; s<60; s++ ) {
try{ Thread.sleep(60000);}catch(Exception e){}
System.gc();
}
}
Click to expand...


I have had problems in the past when I didn't close my
HttpURLConnection. You might try that.
 
A

Alan Krueger

Knute said:
(e-mail address removed) wrote: [snip 350-some quoted lines]

I have had problems in the past when I didn't close my
HttpURLConnection. You might try that.
Click to expand...

Please try to trim quoted text before replying. Quoting hundreds of
lines of text isn't really necessary to give context for a 1-2 line
response.
 
K

Kevin McMurtrie

Your code isn't even close to complete. The interesting parts, like
what you do with the input stream, is missing.

Generally, temporary hangs and memory leaks are caused by not completing
the streams or abandoning HTTP 1.1 connections without specifying
"Keep-Alive: close" in the header.
 
T

timjowers

Thanks for everyone's feedback. I determined
1) closing input stream speeds up memory recovery (closeAll below)
2) setting references to null speeds up memory recovery (zeroMem below)
3) calling gc speeds up memory recovery. Of course, at a cost but if
mem is the over-arching problem as in my case then this cost may be
necessary. (System.gc, System.gc in thread class below)

I made a very simple example below (compiles as-is :) and similar to
my real code which has lots of threads. I think these general memory
usage concepts will be applicable to other areas and get the
OutOfMemoryError under control. Also changed startup setting for Java.

Thanks for the feedback,
TimJowers

package WebGen;
/**
* Sample memory usage for URLConnection
*/
import java.net.*;
import java.io.*;



public class Leaky {

private URL url=null;
private URLConnection urlConnection=null;
private InputStream is=null;
private StringBuffer sbuf = null;
private byte[] bytesRead = new byte[4096];

protected void zeroMem()
{
url=null;
urlConnection=null;
is=null;
sbuf = null;
bytesRead = null;
}

public boolean doWget ( String httpRequest ) throws Exception {

System.out.println(httpRequest);

try {
url = new URL(httpRequest);
} catch (MalformedURLException e) {
throw new Exception("Malformed URL, must be
http://somehost[/file]");
}
try {
urlConnection = url.openConnection();
} catch (IOException e) {
throw new Exception("Error opening URL Connection to " +
httpRequest);
}

DataOutputStream printout;
DataInputStream input;

urlConnection.setDoInput (true);
urlConnection.setUseCaches (false);
// may need to set header "Keep-Alive: close"
urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
urlConnection.setRequestProperty("Cache-Control","no-cache");

if( !readHttp() )
return false;

closeAll();

return true;
}

class TestLeaky extends Thread
{
String httpRequest;
public void run() {
Leaky wget = new Leaky();
try{
wget.doWget( httpRequest );
String result = wget.getOutput();
System.out.println( httpRequest + " page has " + result.length() +
" characters" );
wget.zeroMem(); // actually speeds gc
wget=null;
System.gc(); // probably overkill but best way to minimize memory
usage
System.gc();
} catch( Exception e) {
System.err.println( e );
}
}
}

/** Main invoked from the prompt.
* @param args Arguments from the Stdin
* @throws Exception Throws Exception
*/
static String[] urls = {"http://www.youart.net/",
"http://www.microsoft.net/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.ncr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.ebay.com/", "http://www.yahoo.com/",
"http://www.googlr.com/", "http://www.ibm.com/",
"http://www.ford.com/", "http://www.cisco.com/",
"http://www.honda.com/", "http://www.toyota.com/",
"http://www.talkbackforum.com/", "http://www.unitedswe.com/",
"http://www.nbc.com/", "http://www.abc.com/",
"http://www.cbs.com/", "http://www.bbc.com/",
"http://www.qbert.com/", "http://www.yahoo.com/",
"http://www.starwars.com/", "http://www.nasa.gov/",
"http://www.startrek.com/", "http://www.cisco.com/",
"http://www.starman.com/", "http://www.mars.com/",
"http://www.cnn.com/"};
public static void main ( String[] args ) throws Exception {

Leaky lek = new Leaky();
for( int e=0; e<urls.length; e++)
{ TestLeaky thrd = lek.new TestLeaky();
thrd.httpRequest = urls[e];
thrd.start();
}

System.gc();
System.gc();
System.gc();
// memory is leaked at this point. Next memory is magically returned
if we sleep awhile.
for( int s=0; s<60; s++ ) {
try{ Thread.sleep(5000);}catch(Exception e){}
System.gc();
}
}

public Leaky() { }

private void closeAll() {
if( urlConnection != null )
try {
urlConnection.getInputStream().close();
if( urlConnection.getDoOutput() == true )
urlConnection.getOutputStream().close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
urlConnection = null;
url = null;
is = null;
//System.gc();
}
private boolean readHttp() throws Exception {
try {
sbuf = new StringBuffer();
is = urlConnection.getInputStream();
if (urlConnection.getHeaderField("Set-Cookie") != null) {
System.out.println(urlConnection.getHeaderField("Set-Cookie"));
sbuf.append(urlConnection.getHeaderField("Set-Cookie"));
}
int amount=0;
System.out.println("Connected, retrieving file...");
while ((amount = is.read(bytesRead)) > 0) {
//System.out.println( "Just read bytes " + amount );
for (int i=0; i<amount; i++) {
sbuf.append((char)bytesRead);
}
}
} catch (IOException e) {
System.err.println( e );
return false;
}

try {
is.close();
} catch (IOException e) {
System.err.println("Error closing connection.");
return false;
}
return true;
}

/** Returns the String returned by the Operation, like the content of
a Web page.
* @return Returns a String object containing the received information
*/
public String getOutput() {
if( sbuf == null ) return "";
String output = sbuf.toString();
//sbuf.delete(0,sbuf.length());
//sbuf = null;
return output;
}
}
 
K

Knute Johnson

Alan said:
Knute said:
(e-mail address removed) wrote:
Click to expand...

[snip 350-some quoted lines]
I have had problems in the past when I didn't close my
HttpURLConnection. You might try that.
Click to expand...


Please try to trim quoted text before replying. Quoting hundreds of
lines of text isn't really necessary to give context for a 1-2 line
response.
Click to expand...

bite me!
 
T

Thomas Weidenfeller

Thanks for everyone's feedback. I determined
1) closing input stream speeds up memory recovery (closeAll below)
2) setting references to null speeds up memory recovery (zeroMem below)
3) calling gc speeds up memory recovery. Of course, at a cost but if
mem is the over-arching problem as in my case then this cost may be
necessary. (System.gc, System.gc in thread class below)
Click to expand...

2) is seldom needed (only if the method will hold onto the reference for
a long time after the object is needed). 3) is usually not needed at
all. The JVM will run the GC when it becomes low on memory. So when you
see an out of memory error, the JVM really has no memory, and did
previously try "everything" to gain free memory. A manual attempt to run
the GC is no different from a JVM's automatic attempt to run the GC.

Running the GC manually can not clean up any references you accidentally
hold on to. The GC can't fix coding errors. Get a memory profiler to
figure out who holds the references. Manually calling the GC just
needlessly burns CPU cycles.

/Thomas
 
C

Chris Uppal

Thomas said:
3) is usually not needed at
all. The JVM will run the GC when it becomes low on memory. So when you
see an out of memory error, the JVM really has no memory, and did
previously try "everything" to gain free memory. A manual attempt to run
the GC is no different from a JVM's automatic attempt to run the GC.
Click to expand...

The point is that if some external resource ends up being managed (whether by
design or by accident) by finalisation (or reference queues, etc), then manual
GC may allow the finalisation process to 'see' the no-longer-needed resources
and clean them up /before/ the JVM realises that it needs to reclaim memory.
Not a good solution, and one that is never /guaranteed/ to work, but it can
help in some circumstances.

-- chris
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

URLConnection 6
URLConnection cannot flush 2
The distinction between a java applet and an application 1
URLConnection & Http/Https 0
Cannot retrieve the Web document 1
Implementing Many Stacks in the Same Program 1
SSCCE of my graphical interface with memory leak 9
user UrlConnection to upload multipart 3

Members online

No members online now.
Total: 35 (members: 0, guests: 35)
Robots: 454

Forum statistics

Threads
473,768
Messages
2,569,574
Members
45,048
Latest member
verona

Latest Threads

Top