URLConnection "memory leak"

Discussion in 'Java' started by timjowers@gmail.com, Jun 3, 2005.

  1. Guest

    URLConnection leaks memory and after a while will return the memory. No
    amount of calling System.gc(); will clean it up. OK for small apps but
    I am writing one that downloads 100 or so pages synchronously. OK
    unless I experience several clients asking my server to do this at
    once. Then Java gives the almost useless OutOfMemoryError. So much for
    Garbage Collection. At this point GC just means random
    non-deterministic memory management and means, unlike in C++, I cannot
    achieve a robust server application. How can Java justify giving
    OutOfMemoryError when NO REFERENCES ARE HELD IN MY PROGRAM!!!! I guess
    URLConnection must have some internal static buggy code.

    Any ideas how to get URLConnection to stop hanging onto memory?

    Here's some test code below.

    TimJowers

    /**
    * Sample memory
    */
    import java.net.*;
    import java.io.*;
    import java.util.*;
    import java.security.*;



    public class Leaky {

    public String strXML = null;
    private String proxyHost=null, proxyPort=null, username=null,
    password=null,
    httpRequest=null, base64Encoded=null, proxyType="2",
    formValues=null, formMethod="", optionFile=null;
    private URL url=null;
    private URLConnection urlConnection=null;
    private InputStream is=null;
    private File file=null;
    private FileOutputStream out=null;
    private int ftpPort=21;
    private boolean notAssigned = true,
    ftpTrue=false, ftpPassive=false, ftpAscii=false,
    https=false, nonverbose=true, post=false, booleanOptionFile=false;
    private StringBuffer sbuf = null;
    private byte[] bytesRead = new byte[4096];

    protected void zeroMem()
    {
    strXML = null;
    proxyHost=null; proxyPort=null; username=null; password=null;
    httpRequest=null; base64Encoded=null;
    formValues=null;
    formMethod=""; optionFile=null;
    url=null;
    urlConnection=null;
    is=null;
    file=null;
    out=null;
    sbuf = null;
    bytesRead = null;
    }

    public URLConnection setupProxy ( String [] args ) throws Exception
    {
    getOptions(args);

    if (https) {
    if (!nonverbose) {
    System.out.println("Using https communication");
    }
    Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
    System.setProperty("http.keepAlive", "false");
    System.setProperty( "java.protocol.handler.pkgs",
    "com.sun.net.ssl.internal.www.protocol" );
    }
    if (formMethod != null && formMethod.equalsIgnoreCase("get")) {
    httpRequest = httpRequest + "?" + encodeHttpString(formValues);
    }
    try {
    url = new URL(httpRequest);
    } catch (MalformedURLException e) {
    if (!nonverbose) {
    System.err.println("Malformed URL, must be
    http://somehost[/file]");
    } else {
    throw new Exception("Malformed URL, must be
    http://somehost[/file]");
    }
    return null;
    }
    if (proxyHost != null) {
    if (proxyType.equals("1") || proxyType.equals("3")) {
    System.getProperties().put( "proxySet", "true" );
    System.getProperties().put( "proxyHost", proxyHost);
    System.getProperties().put( "proxyPort", proxyPort);
    } else {
    if (proxyType.equals("4")) {
    System.getProperties().put( "socksProxySet", "true");
    System.getProperties().put( "socksProxyHost", proxyHost);
    System.getProperties().put( "SocksProxyPort" , proxyPort);
    } else {
    if (proxyType.equals("5")) {
    System.getProperties().put( "ftpProxySet", "true" );
    System.getProperties().put( "ftpProxyHost", proxyHost );
    System.getProperties().put( "ftpProxyPort", proxyPort );
    } else {
    System.getProperties().put("firewallSet", "true");
    System.getProperties().put("firewallHost", proxyHost);
    System.getProperties().put("firewallPort", proxyPort);
    System.getProperties().put("http.proxyHost", proxyHost);
    System.getProperties().put("http.proxyPort", proxyPort);
    }}}
    if (https && proxyHost != null) {
    System.getProperties().put("https.proxyHost", proxyHost);
    System.getProperties().put("https.proxyPort", proxyPort);
    }
    }
    try {
    urlConnection = url.openConnection();
    } catch (IOException e) {
    if (!nonverbose) {
    System.err.println("Error opening URL connection to " +
    httpRequest);
    } else {
    throw new Exception("Error opening URL Connection to " +
    httpRequest);
    }
    return null;
    }
    if (proxyHost != null) {
    Base64 base64 = new Base64(username + ":" + password);
    base64.encode();
    base64Encoded = "Basic " + base64.getOutgoing();
    urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
    if (proxyType.equals("3")) {
    urlConnection.setRequestProperty("Authorization",
    base64Encoded);
    } else {
    urlConnection.setRequestProperty("Proxy-Authorization",
    base64Encoded);
    }
    }
    return urlConnection;
    }

    public boolean doWget ( String args ) throws Exception {
    // if args are all on one string then break into an array of strings
    StringTokenizer strTok = new StringTokenizer( args );
    String []argv = new String[ strTok.countTokens() ];
    for(int i=0; strTok.hasMoreTokens(); i++ )
    argv = strTok.nextToken();
    return doWget( argv );
    }

    public boolean doWget ( String [] args ) throws Exception {

    System.out.println(httpRequest);
    System.out.println(proxyHost);
    System.out.println(ftpPort);

    urlConnection = setupProxy ( args );

    if (!nonverbose) {
    System.out.println("Connecting to " + httpRequest + "...");
    if(
    0==httpRequest.compareTo("http://www.p2p.wrox.com/listindex.asp") ) //
    hangs
    return false;
    if(
    0==httpRequest.compareTo("http://www.soton.ac.uk/~chst/direct.htm") )
    // hangs
    return false;
    if(
    0==httpRequest.compareTo("https://www.ahamembership.com/assoc6/k.cgi?p=10-TX&acct_code=TX004-10TX-T")
    ) // security exception
    return false;
    if(
    0==httpRequest.compareTo("https://www25.hway.net/onl261/cgi-local/sgx/shop.cgi?page=order.html&afnum=21")
    ) // security exception
    return false;
    if( 0==httpRequest.compareTo("https://grc.com/x/ne.dll?bh0bkyd2") )
    // security exception
    return false;
    }
    try
    {
    DataOutputStream printout;
    DataInputStream input;

    urlConnection.setDoInput (true);
    urlConnection.setUseCaches (false);
    urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
    urlConnection.setRequestProperty("Cache-Control","no-cache");

    if (null != strXML) // add XML data as HTTP payload
    { System.out.println("Adding XML...");
    // Send POST output.
    printout = new DataOutputStream (urlConnection.getOutputStream ());
    String content =
    "CoNUM=" + URLEncoder.encode ("2") +
    "&PASSWORD=" + URLEncoder.encode ("1");
    String msg;
    msg = "<?xml version=" + "\"" + "1.0" + "\"" + "?>";
    msg = msg + "</XML>";
    content = msg;
    System.out.println( " --- XML to Web Server ---\r\r" + content );
    printout.writeBytes (content);
    printout.flush ();
    printout.close ();
    }
    }
    catch (MalformedURLException me)
    {
    System.err.println("MalformedURLException: " + me);
    }
    catch (IOException ioe)
    {
    System.err.println("IOException: " + ioe.getMessage());
    }
    if( !readHttp() )
    return false;
    closeAll();
    return true;
    }

    private void getOptions( String[] args ) throws Exception {
    for (int i=0; i<args.length; i++) {
    if (args.equals("-h")) {
    StringTokenizer st = new StringTokenizer(args[i+1], ":");
    int count=0;
    while (st.hasMoreTokens()) {
    if (count == 0 ) {
    proxyHost=st.nextToken();
    }
    if (count == 1) {
    proxyPort=st.nextToken();
    }
    count++;
    }
    if (proxyPort == null) {
    proxyPort = "80";
    }
    i++;
    notAssigned=false;
    }
    if (args.equals("-u")) {
    username=args[i+1];
    i++;
    notAssigned=false;
    }
    if (args.equals("-p")) {
    password=args[i+1];
    i++;
    notAssigned=false;
    }
    if (args.equals("-t")) {
    proxyType=args[i+1];
    i++;
    notAssigned=false;
    }
    if (args.equals("-post")) {
    formValues=args[i+1];
    i++;
    notAssigned=false;
    }
    if (args.equals("-method")) {
    formMethod=args[i+1];
    i++;
    notAssigned=false;
    }
    if( args.equals("-xml") ) // get XML file
    { strXML = new String("<XML></XML>");
    }
    if (proxyHost != null && ftpTrue==true) {
    if (!nonverbose) {
    System.err.println("Proxy and Ftp cannot coexist use -h or -ftp");
    } else {
    throw new Exception("Proxy and Ftp cannot coexist use -h or
    -ftp");
    }
    return;
    }
    if (notAssigned) {
    httpRequest=args;
    if (httpRequest.substring(0, 5).equalsIgnoreCase("https")) {
    https=true;
    }
    }
    notAssigned=true;
    }
    }

    /** Main invoked from the prompt.
    * @param args Arguments from the Stdin
    * @throws Exception Throws Exception
    */
    static String[] urls = {"http://www.youart.net/",
    "http://www.microsoft.net/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.cnn.com/"};
    public static void main ( String[] args ) throws Exception {

    for( int e=0; e<urls.length; e++)
    { Leaky wget = new Leaky();
    wget.doWget( urls[e] );
    String result = wget.getOutput();
    System.out.println( "page has " + result.length() + " characters" );
    wget.zeroMem();
    wget=null;
    System.gc();
    System.gc();
    System.gc();
    }
    // memory is leaked at this point. Next memory is magically returned
    if we sleep awhile.
    for( int s=0; s<60; s++ ) {
    try{ Thread.sleep(60000);}catch(Exception e){}
    System.gc();
    }
    }
    , Jun 3, 2005
    #1
    1. Advertising

  2. On 2 Jun 2005 21:51:37 -0700, wrote:

    ....
    > public String strXML = null;


    Please refrain form posting tab characters to usenet, as they are
    often expanded to ridiculous lengths by news reader software. And
    keep your line lengths short. Otherwise lines wrap, and therefore break..
    ....
    > System.err.println("Malformed URL, must be
    > http://somehost[/file]");


    Also, you are missing a closing '}'. If you had stuck to a single
    convention for formatting code, this might be easier to spot, but it
    seems you are mixing and matching..

    When those problems are fixed, I get..

    C:\..\Leaky.java:60: cannot find symbol
    symbol : method encodeHttpString(java.lang.String)
    location: class Leaky
    httpRequest = httpRequest + "?" +
    encodeHttpString(formValues);
    ^
    C:\..\Leaky.java:114: cannot find symbol
    symbol : class Base64
    location: class Leaky
    Base64 base64 = new Base64(username + ":" +
    password);
    ^
    C:\..\Leaky.java:114: cannot find symbol
    symbol : class Base64
    location: class Leaky
    Base64 base64 = new Base64(username + ":" +
    password);
    ^
    C:\..\Leaky.java:203: cannot find symbol
    symbol : method readHttp()
    location: class Leaky
    if( !readHttp() )
    ^
    C:\..\Leaky.java:205: cannot find symbol
    symbol : method closeAll()
    location: class Leaky
    closeAll();
    ^
    C:\..\Leaky.java:321: cannot find symbol
    symbol : method getOutput()
    location: class Leaky
    String result = wget.getOutput();
    ^
    Note: C:\..\Leaky.java uses or overrides a deprecated API.
    Note: Recompile with -Xlint:deprecation for details.
    6 errors

    ....It seems you might benefit from a document specifically prepared
    to help people prepare a good example of a problem. I suggest
    (and recommend*) this one. <http://www.physci.org/codes/sscce.jsp>

    * Not surprising, since I wrote and host it.

    As far as the actual problem, I have no insights. I was curious
    to see your code break, but do not have a great deal of experience
    with URLConnection (at least, not in terms of memory leaks).

    --
    Andrew Thompson
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.PhySci.org/ Open-source software suite
    http://www.1point1C.org/ Science & Technology
    http://www.LensEscapes.com/ Images that escape the mundane
    Andrew Thompson, Jun 3, 2005
    #2
    1. Advertising

  3. wrote:
    > URLConnection leaks memory and after a while will return the memory. No
    > amount of calling System.gc(); will clean it up. OK for small apps but
    > I am writing one that downloads 100 or so pages synchronously. OK
    > unless I experience several clients asking my server to do this at
    > once. Then Java gives the almost useless OutOfMemoryError. So much for
    > Garbage Collection. At this point GC just means random
    > non-deterministic memory management and means, unlike in C++, I cannot
    > achieve a robust server application. How can Java justify giving
    > OutOfMemoryError when NO REFERENCES ARE HELD IN MY PROGRAM!!!! I guess
    > URLConnection must have some internal static buggy code.
    >
    > Any ideas how to get URLConnection to stop hanging onto memory?
    >
    > Here's some test code below.
    >
    > TimJowers
    >
    > /**
    > * Sample memory
    > */
    > import java.net.*;
    > import java.io.*;
    > import java.util.*;
    > import java.security.*;
    >
    >
    >
    > public class Leaky {
    >
    > public String strXML = null;
    > private String proxyHost=null, proxyPort=null, username=null,
    > password=null,
    > httpRequest=null, base64Encoded=null, proxyType="2",
    > formValues=null, formMethod="", optionFile=null;
    > private URL url=null;
    > private URLConnection urlConnection=null;
    > private InputStream is=null;
    > private File file=null;
    > private FileOutputStream out=null;
    > private int ftpPort=21;
    > private boolean notAssigned = true,
    > ftpTrue=false, ftpPassive=false, ftpAscii=false,
    > https=false, nonverbose=true, post=false, booleanOptionFile=false;
    > private StringBuffer sbuf = null;
    > private byte[] bytesRead = new byte[4096];
    >
    > protected void zeroMem()
    > {
    > strXML = null;
    > proxyHost=null; proxyPort=null; username=null; password=null;
    > httpRequest=null; base64Encoded=null;
    > formValues=null;
    > formMethod=""; optionFile=null;
    > url=null;
    > urlConnection=null;
    > is=null;
    > file=null;
    > out=null;
    > sbuf = null;
    > bytesRead = null;
    > }
    >
    > public URLConnection setupProxy ( String [] args ) throws Exception
    > {
    > getOptions(args);
    >
    > if (https) {
    > if (!nonverbose) {
    > System.out.println("Using https communication");
    > }
    > Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
    > System.setProperty("http.keepAlive", "false");
    > System.setProperty( "java.protocol.handler.pkgs",
    > "com.sun.net.ssl.internal.www.protocol" );
    > }
    > if (formMethod != null && formMethod.equalsIgnoreCase("get")) {
    > httpRequest = httpRequest + "?" + encodeHttpString(formValues);
    > }
    > try {
    > url = new URL(httpRequest);
    > } catch (MalformedURLException e) {
    > if (!nonverbose) {
    > System.err.println("Malformed URL, must be
    > http://somehost[/file]");
    > } else {
    > throw new Exception("Malformed URL, must be
    > http://somehost[/file]");
    > }
    > return null;
    > }
    > if (proxyHost != null) {
    > if (proxyType.equals("1") || proxyType.equals("3")) {
    > System.getProperties().put( "proxySet", "true" );
    > System.getProperties().put( "proxyHost", proxyHost);
    > System.getProperties().put( "proxyPort", proxyPort);
    > } else {
    > if (proxyType.equals("4")) {
    > System.getProperties().put( "socksProxySet", "true");
    > System.getProperties().put( "socksProxyHost", proxyHost);
    > System.getProperties().put( "SocksProxyPort" , proxyPort);
    > } else {
    > if (proxyType.equals("5")) {
    > System.getProperties().put( "ftpProxySet", "true" );
    > System.getProperties().put( "ftpProxyHost", proxyHost );
    > System.getProperties().put( "ftpProxyPort", proxyPort );
    > } else {
    > System.getProperties().put("firewallSet", "true");
    > System.getProperties().put("firewallHost", proxyHost);
    > System.getProperties().put("firewallPort", proxyPort);
    > System.getProperties().put("http.proxyHost", proxyHost);
    > System.getProperties().put("http.proxyPort", proxyPort);
    > }}}
    > if (https && proxyHost != null) {
    > System.getProperties().put("https.proxyHost", proxyHost);
    > System.getProperties().put("https.proxyPort", proxyPort);
    > }
    > }
    > try {
    > urlConnection = url.openConnection();
    > } catch (IOException e) {
    > if (!nonverbose) {
    > System.err.println("Error opening URL connection to " +
    > httpRequest);
    > } else {
    > throw new Exception("Error opening URL Connection to " +
    > httpRequest);
    > }
    > return null;
    > }
    > if (proxyHost != null) {
    > Base64 base64 = new Base64(username + ":" + password);
    > base64.encode();
    > base64Encoded = "Basic " + base64.getOutgoing();
    > urlConnection.setRequestProperty("Proxy-Connection","Keep-Alive");
    > if (proxyType.equals("3")) {
    > urlConnection.setRequestProperty("Authorization",
    > base64Encoded);
    > } else {
    > urlConnection.setRequestProperty("Proxy-Authorization",
    > base64Encoded);
    > }
    > }
    > return urlConnection;
    > }
    >
    > public boolean doWget ( String args ) throws Exception {
    > // if args are all on one string then break into an array of strings
    > StringTokenizer strTok = new StringTokenizer( args );
    > String []argv = new String[ strTok.countTokens() ];
    > for(int i=0; strTok.hasMoreTokens(); i++ )
    > argv = strTok.nextToken();
    > return doWget( argv );
    > }
    >
    > public boolean doWget ( String [] args ) throws Exception {
    >
    > System.out.println(httpRequest);
    > System.out.println(proxyHost);
    > System.out.println(ftpPort);
    >
    > urlConnection = setupProxy ( args );
    >
    > if (!nonverbose) {
    > System.out.println("Connecting to " + httpRequest + "...");
    > if(
    > 0==httpRequest.compareTo("http://www.p2p.wrox.com/listindex.asp") ) //
    > hangs
    > return false;
    > if(
    > 0==httpRequest.compareTo("http://www.soton.ac.uk/~chst/direct.htm") )
    > // hangs
    > return false;
    > if(
    > 0==httpRequest.compareTo("https://www.ahamembership.com/assoc6/k.cgi?p=10-TX&acct_code=TX004-10TX-T")
    > ) // security exception
    > return false;
    > if(
    > 0==httpRequest.compareTo("https://www25.hway.net/onl261/cgi-local/sgx/shop.cgi?page=order.html&afnum=21")
    > ) // security exception
    > return false;
    > if( 0==httpRequest.compareTo("https://grc.com/x/ne.dll?bh0bkyd2") )
    > // security exception
    > return false;
    > }
    > try
    > {
    > DataOutputStream printout;
    > DataInputStream input;
    >
    > urlConnection.setDoInput (true);
    > urlConnection.setUseCaches (false);
    > urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
    > urlConnection.setRequestProperty("Cache-Control","no-cache");
    >
    > if (null != strXML) // add XML data as HTTP payload
    > { System.out.println("Adding XML...");
    > // Send POST output.
    > printout = new DataOutputStream (urlConnection.getOutputStream ());
    > String content =
    > "CoNUM=" + URLEncoder.encode ("2") +
    > "&PASSWORD=" + URLEncoder.encode ("1");
    > String msg;
    > msg = "<?xml version=" + "\"" + "1.0" + "\"" + "?>";
    > msg = msg + "</XML>";
    > content = msg;
    > System.out.println( " --- XML to Web Server ---\r\r" + content );
    > printout.writeBytes (content);
    > printout.flush ();
    > printout.close ();
    > }
    > }
    > catch (MalformedURLException me)
    > {
    > System.err.println("MalformedURLException: " + me);
    > }
    > catch (IOException ioe)
    > {
    > System.err.println("IOException: " + ioe.getMessage());
    > }
    > if( !readHttp() )
    > return false;
    > closeAll();
    > return true;
    > }
    >
    > private void getOptions( String[] args ) throws Exception {
    > for (int i=0; i<args.length; i++) {
    > if (args.equals("-h")) {
    > StringTokenizer st = new StringTokenizer(args[i+1], ":");
    > int count=0;
    > while (st.hasMoreTokens()) {
    > if (count == 0 ) {
    > proxyHost=st.nextToken();
    > }
    > if (count == 1) {
    > proxyPort=st.nextToken();
    > }
    > count++;
    > }
    > if (proxyPort == null) {
    > proxyPort = "80";
    > }
    > i++;
    > notAssigned=false;
    > }
    > if (args.equals("-u")) {
    > username=args[i+1];
    > i++;
    > notAssigned=false;
    > }
    > if (args.equals("-p")) {
    > password=args[i+1];
    > i++;
    > notAssigned=false;
    > }
    > if (args.equals("-t")) {
    > proxyType=args[i+1];
    > i++;
    > notAssigned=false;
    > }
    > if (args.equals("-post")) {
    > formValues=args[i+1];
    > i++;
    > notAssigned=false;
    > }
    > if (args.equals("-method")) {
    > formMethod=args[i+1];
    > i++;
    > notAssigned=false;
    > }
    > if( args.equals("-xml") ) // get XML file
    > { strXML = new String("<XML></XML>");
    > }
    > if (proxyHost != null && ftpTrue==true) {
    > if (!nonverbose) {
    > System.err.println("Proxy and Ftp cannot coexist use -h or -ftp");
    > } else {
    > throw new Exception("Proxy and Ftp cannot coexist use -h or
    > -ftp");
    > }
    > return;
    > }
    > if (notAssigned) {
    > httpRequest=args;
    > if (httpRequest.substring(0, 5).equalsIgnoreCase("https")) {
    > https=true;
    > }
    > }
    > notAssigned=true;
    > }
    > }
    >
    > /** Main invoked from the prompt.
    > * @param args Arguments from the Stdin
    > * @throws Exception Throws Exception
    > */
    > static String[] urls = {"http://www.youart.net/",
    > "http://www.microsoft.net/",
    > "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    > "http://www.nbc.com/", "http://www.abc.com/",
    > "http://www.cbs.com/", "http://www.bbc.com/",
    > "http://www.ebay.com/", "http://www.yahoo.com/",
    > "http://www.ncr.com/", "http://www.ibm.com/",
    > "http://www.ford.com/", "http://www.cisco.com/",
    > "http://www.honda.com/", "http://www.toyota.com/",
    > "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    > "http://www.nbc.com/", "http://www.abc.com/",
    > "http://www.cbs.com/", "http://www.bbc.com/",
    > "http://www.ebay.com/", "http://www.yahoo.com/",
    > "http://www.ncr.com/", "http://www.ibm.com/",
    > "http://www.ford.com/", "http://www.cisco.com/",
    > "http://www.honda.com/", "http://www.toyota.com/",
    > "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    > "http://www.nbc.com/", "http://www.abc.com/",
    > "http://www.cbs.com/", "http://www.bbc.com/",
    > "http://www.ebay.com/", "http://www.yahoo.com/",
    > "http://www.ncr.com/", "http://www.ibm.com/",
    > "http://www.ford.com/", "http://www.cisco.com/",
    > "http://www.honda.com/", "http://www.toyota.com/",
    > "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    > "http://www.nbc.com/", "http://www.abc.com/",
    > "http://www.cbs.com/", "http://www.bbc.com/",
    > "http://www.ebay.com/", "http://www.yahoo.com/",
    > "http://www.ncr.com/", "http://www.ibm.com/",
    > "http://www.ford.com/", "http://www.cisco.com/",
    > "http://www.honda.com/", "http://www.toyota.com/",
    > "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    > "http://www.nbc.com/", "http://www.abc.com/",
    > "http://www.cbs.com/", "http://www.bbc.com/",
    > "http://www.ebay.com/", "http://www.yahoo.com/",
    > "http://www.ncr.com/", "http://www.ibm.com/",
    > "http://www.ford.com/", "http://www.cisco.com/",
    > "http://www.honda.com/", "http://www.toyota.com/",
    > "http://www.cnn.com/"};
    > public static void main ( String[] args ) throws Exception {
    >
    > for( int e=0; e<urls.length; e++)
    > { Leaky wget = new Leaky();
    > wget.doWget( urls[e] );
    > String result = wget.getOutput();
    > System.out.println( "page has " + result.length() + " characters" );
    > wget.zeroMem();
    > wget=null;
    > System.gc();
    > System.gc();
    > System.gc();
    > }
    > // memory is leaked at this point. Next memory is magically returned
    > if we sleep awhile.
    > for( int s=0; s<60; s++ ) {
    > try{ Thread.sleep(60000);}catch(Exception e){}
    > System.gc();
    > }
    > }
    >


    I have had problems in the past when I didn't close my
    HttpURLConnection. You might try that.

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Jun 3, 2005
    #3
  4. Alan Krueger Guest

    Knute Johnson wrote:
    > wrote:

    [snip 350-some quoted lines]
    >
    > I have had problems in the past when I didn't close my
    > HttpURLConnection. You might try that.


    Please try to trim quoted text before replying. Quoting hundreds of
    lines of text isn't really necessary to give context for a 1-2 line
    response.
    Alan Krueger, Jun 4, 2005
    #4
  5. Your code isn't even close to complete. The interesting parts, like
    what you do with the input stream, is missing.

    Generally, temporary hangs and memory leaks are caused by not completing
    the streams or abandoning HTTP 1.1 connections without specifying
    "Keep-Alive: close" in the header.
    Kevin McMurtrie, Jun 4, 2005
    #5
  6. Guest

    Thanks for everyone's feedback. I determined
    1) closing input stream speeds up memory recovery (closeAll below)
    2) setting references to null speeds up memory recovery (zeroMem below)
    3) calling gc speeds up memory recovery. Of course, at a cost but if
    mem is the over-arching problem as in my case then this cost may be
    necessary. (System.gc, System.gc in thread class below)

    I made a very simple example below (compiles as-is :) and similar to
    my real code which has lots of threads. I think these general memory
    usage concepts will be applicable to other areas and get the
    OutOfMemoryError under control. Also changed startup setting for Java.

    Thanks for the feedback,
    TimJowers

    package WebGen;
    /**
    * Sample memory usage for URLConnection
    */
    import java.net.*;
    import java.io.*;



    public class Leaky {

    private URL url=null;
    private URLConnection urlConnection=null;
    private InputStream is=null;
    private StringBuffer sbuf = null;
    private byte[] bytesRead = new byte[4096];

    protected void zeroMem()
    {
    url=null;
    urlConnection=null;
    is=null;
    sbuf = null;
    bytesRead = null;
    }

    public boolean doWget ( String httpRequest ) throws Exception {

    System.out.println(httpRequest);

    try {
    url = new URL(httpRequest);
    } catch (MalformedURLException e) {
    throw new Exception("Malformed URL, must be
    http://somehost[/file]");
    }
    try {
    urlConnection = url.openConnection();
    } catch (IOException e) {
    throw new Exception("Error opening URL Connection to " +
    httpRequest);
    }

    DataOutputStream printout;
    DataInputStream input;

    urlConnection.setDoInput (true);
    urlConnection.setUseCaches (false);
    // may need to set header "Keep-Alive: close"
    urlConnection.setRequestProperty("Accept","text/xml,text/*,text/html");
    urlConnection.setRequestProperty("Cache-Control","no-cache");

    if( !readHttp() )
    return false;

    closeAll();

    return true;
    }

    class TestLeaky extends Thread
    {
    String httpRequest;
    public void run() {
    Leaky wget = new Leaky();
    try{
    wget.doWget( httpRequest );
    String result = wget.getOutput();
    System.out.println( httpRequest + " page has " + result.length() +
    " characters" );
    wget.zeroMem(); // actually speeds gc
    wget=null;
    System.gc(); // probably overkill but best way to minimize memory
    usage
    System.gc();
    } catch( Exception e) {
    System.err.println( e );
    }
    }
    }

    /** Main invoked from the prompt.
    * @param args Arguments from the Stdin
    * @throws Exception Throws Exception
    */
    static String[] urls = {"http://www.youart.net/",
    "http://www.microsoft.net/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.ncr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.ebay.com/", "http://www.yahoo.com/",
    "http://www.googlr.com/", "http://www.ibm.com/",
    "http://www.ford.com/", "http://www.cisco.com/",
    "http://www.honda.com/", "http://www.toyota.com/",
    "http://www.talkbackforum.com/", "http://www.unitedswe.com/",
    "http://www.nbc.com/", "http://www.abc.com/",
    "http://www.cbs.com/", "http://www.bbc.com/",
    "http://www.qbert.com/", "http://www.yahoo.com/",
    "http://www.starwars.com/", "http://www.nasa.gov/",
    "http://www.startrek.com/", "http://www.cisco.com/",
    "http://www.starman.com/", "http://www.mars.com/",
    "http://www.cnn.com/"};
    public static void main ( String[] args ) throws Exception {

    Leaky lek = new Leaky();
    for( int e=0; e<urls.length; e++)
    { TestLeaky thrd = lek.new TestLeaky();
    thrd.httpRequest = urls[e];
    thrd.start();
    }

    System.gc();
    System.gc();
    System.gc();
    // memory is leaked at this point. Next memory is magically returned
    if we sleep awhile.
    for( int s=0; s<60; s++ ) {
    try{ Thread.sleep(5000);}catch(Exception e){}
    System.gc();
    }
    }

    public Leaky() { }

    private void closeAll() {
    if( urlConnection != null )
    try {
    urlConnection.getInputStream().close();
    if( urlConnection.getDoOutput() == true )
    urlConnection.getOutputStream().close();
    } catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    urlConnection = null;
    url = null;
    is = null;
    //System.gc();
    }
    private boolean readHttp() throws Exception {
    try {
    sbuf = new StringBuffer();
    is = urlConnection.getInputStream();
    if (urlConnection.getHeaderField("Set-Cookie") != null) {
    System.out.println(urlConnection.getHeaderField("Set-Cookie"));
    sbuf.append(urlConnection.getHeaderField("Set-Cookie"));
    }
    int amount=0;
    System.out.println("Connected, retrieving file...");
    while ((amount = is.read(bytesRead)) > 0) {
    //System.out.println( "Just read bytes " + amount );
    for (int i=0; i<amount; i++) {
    sbuf.append((char)bytesRead);
    }
    }
    } catch (IOException e) {
    System.err.println( e );
    return false;
    }

    try {
    is.close();
    } catch (IOException e) {
    System.err.println("Error closing connection.");
    return false;
    }
    return true;
    }

    /** Returns the String returned by the Operation, like the content of
    a Web page.
    * @return Returns a String object containing the received information
    */
    public String getOutput() {
    if( sbuf == null ) return "";
    String output = sbuf.toString();
    //sbuf.delete(0,sbuf.length());
    //sbuf = null;
    return output;
    }
    }
    , Jun 4, 2005
    #6
  7. Alan Krueger wrote:
    > Knute Johnson wrote:
    >
    >> wrote:

    >
    > [snip 350-some quoted lines]
    >
    >>
    >> I have had problems in the past when I didn't close my
    >> HttpURLConnection. You might try that.

    >
    >
    > Please try to trim quoted text before replying. Quoting hundreds of
    > lines of text isn't really necessary to give context for a 1-2 line
    > response.


    bite me!

    --

    Knute Johnson
    email s/nospam/knute/
    Knute Johnson, Jun 5, 2005
    #7
  8. wrote:
    > Thanks for everyone's feedback. I determined
    > 1) closing input stream speeds up memory recovery (closeAll below)
    > 2) setting references to null speeds up memory recovery (zeroMem below)
    > 3) calling gc speeds up memory recovery. Of course, at a cost but if
    > mem is the over-arching problem as in my case then this cost may be
    > necessary. (System.gc, System.gc in thread class below)


    2) is seldom needed (only if the method will hold onto the reference for
    a long time after the object is needed). 3) is usually not needed at
    all. The JVM will run the GC when it becomes low on memory. So when you
    see an out of memory error, the JVM really has no memory, and did
    previously try "everything" to gain free memory. A manual attempt to run
    the GC is no different from a JVM's automatic attempt to run the GC.

    Running the GC manually can not clean up any references you accidentally
    hold on to. The GC can't fix coding errors. Get a memory profiler to
    figure out who holds the references. Manually calling the GC just
    needlessly burns CPU cycles.

    /Thomas

    --
    The comp.lang.java.gui FAQ:
    ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
    Thomas Weidenfeller, Jun 6, 2005
    #8
  9. Chris Uppal Guest

    Thomas Weidenfeller wrote:

    > 3) is usually not needed at
    > all. The JVM will run the GC when it becomes low on memory. So when you
    > see an out of memory error, the JVM really has no memory, and did
    > previously try "everything" to gain free memory. A manual attempt to run
    > the GC is no different from a JVM's automatic attempt to run the GC.


    The point is that if some external resource ends up being managed (whether by
    design or by accident) by finalisation (or reference queues, etc), then manual
    GC may allow the finalisation process to 'see' the no-longer-needed resources
    and clean them up /before/ the JVM realises that it needs to reclaim memory.
    Not a good solution, and one that is never /guaranteed/ to work, but it can
    help in some circumstances.

    -- chris
    Chris Uppal, Jun 6, 2005
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kristoffel

    URLConnection and proxy

    Kristoffel, Jul 27, 2003, in forum: Java
    Replies:
    0
    Views:
    1,563
    Kristoffel
    Jul 27, 2003
  2. Replies:
    2
    Views:
    3,572
  3. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    681
    Eric Sosman
    Mar 22, 2005
  4. cham
    Replies:
    5
    Views:
    757
  5. Mark Probert
    Replies:
    4
    Views:
    318
    Mark Probert
    Feb 9, 2005
Loading...

Share This Page