WebRequest/WebResponse gotcha

[George Ter-Saakov]

Just learned (hard way) one gotcha about working with WebRequest.
Decided to share with with you guys since it's kind of not documented ( at least I did not see it).

Given following code
try
{
HttpWebRequest httpWebRequest = WebRequest.Create(http://www.site.com);
HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
......
httpWebResponse.Close();
}
catch (WebException e)
{
if (e.Response != null)
e.Response.Close();
.......
}

It's important to catch WebException (and not Exception) since it has a Response object. If you do not close it, you will run into problem. Since only 2 connections (default) to the remote server can be open at the same time.

So 2 "500"' errors and your application will not be able to connect to the server anymore until GC runs or it restart...

PS: So far I was always catching Exception and did not even realized that I must catch WebException and properly close WebException.Response until my "hard landing" today when everything stopped working after 2 legitimate 404 (Not found) returned by the server.

Good luck.
George.

 

[John Saunders [MVP]]

Just learned (hard way) one gotcha about working with WebRequest.
Decided to share with with you guys since it's kind of not documented ( at least I did not see it).

Given following code
try
{
HttpWebRequest httpWebRequest = WebRequest.Create(http://www.site.com);
HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
......
httpWebResponse.Close();
}
catch (WebException e)
{
if (e.Response != null)
e.Response.Close();
.......
}


The WebResponse class implements IDisposable. It should be placed in a using statement. For instance:

HttpWebRequest httpWebRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://www.site.com");

using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())

{

using (Stream stream = httpWebResponse.GetResponseStream())

{

using (StreamReader reader = new StreamReader(stream))

{

_textBox1.Text = reader.ReadToEnd();

}

}

}

 

[George Ter-Saakov]

That is correct... I am always doing that... (I simplified my code for clarity).
But it still will not help you.
When GetResponse throws an error the httpWebResponse is null. So you even will not be able to call Dispose() on it.

You must catch WebException and do Dispose on WebException.Response object.



George.
"John Saunders [MVP]" <john.saunders at trizetto.com> wrote in message Just learned (hard way) one gotcha about working with WebRequest.
Decided to share with with you guys since it's kind of not documented ( at least I did not see it).

Given following code
try
{
HttpWebRequest httpWebRequest = WebRequest.Create(http://www.site.com);
HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
......
httpWebResponse.Close();
}
catch (WebException e)
{
if (e.Response != null)
e.Response.Close();
.......
}


The WebResponse class implements IDisposable. It should be placed in a using statement. For instance:

HttpWebRequest httpWebRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://www.site.com");

using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())

{

using (Stream stream = httpWebResponse.GetResponseStream())

{

using (StreamReader reader = new StreamReader(stream))

{

_textBox1.Text = reader.ReadToEnd();

}

}

}


--
--------------------------------------------------------------------------------
John Saunders | MVP - Windows Server System - Connected System Developer

 

[John Saunders [MVP]]

That is correct... I am always doing that... (I simplified my code for clarity).
But it still will not help you.
When GetResponse throws an error the httpWebResponse is null. So you even will not be able to call Dispose() on it.

You must catch WebException and do Dispose on WebException.Response object.

George, are you sure that you need to Dispose it? The documentation doesn't say so, and they've had six years to correct the documentation. The WebResponse in the Response property may be disconnected from the network by the time it gets added to the exception.

 

[George Ter-Saakov]

I am positive...

I just run into it "face down".. I am writing a backup application that is backing up files/folders to S3 Amazon service.
And the way S3 works it's perfectly normal (with my application) to get 404 error which is end up being WebException

Then I noticed that after I get two 404 my application hangs and can not connect to Amazon service anymore (restarting application always fixed the problem).

I am well aware that by default .NET allows only 2 open HTTP connections to the same server so I immediately realized that it's not a coincidence (number 2 here and there).

Well, the HttpWebRequest does not have Dispose (nor Close). So there is nothing to close if WebException is thrown...

So i started to debug a hell out of it.
In debug I noticed that when I catch Exception it's actually WebException object and it has it's own Response stream.
Bingo... Nobody ever closes that stream..

And theoretically this Stream can have any size.... some websites supply the whole "book" if you request non existent page...
In my case it was about 100 bytes but still I had to close it in my catch(..) statement....


PS: I was surprised and disappointed as hell that it's not well documented.. All those years I was writing buggy application. Never actually caught a problem until now...

George.

"John Saunders [MVP]" <john.saunders at trizetto.com> wrote in message That is correct... I am always doing that... (I simplified my code for clarity).
But it still will not help you.
When GetResponse throws an error the httpWebResponse is null. So you even will not be able to call Dispose() on it.

You must catch WebException and do Dispose on WebException.Response object.

George, are you sure that you need to Dispose it? The documentation doesn't say so, and they've had six years to correct the documentation. The WebResponse in the Response property may be disconnected from the network by the time it gets added to the exception.
--
--------------------------------------------------------------------------------
John Saunders | MVP - Windows Server System - Connected System Developer

 

[John Saunders [MVP]]

I am positive...

I just run into it "face down".. I am writing a backup application that is backing up files/folders to S3 Amazon service.
And the way S3 works it's perfectly normal (with my application) to get 404 error which is end up being WebException

Then I noticed that after I get two 404 my application hangs and can not connect to Amazon service anymore (restarting application always fixed the problem).

I am well aware that by default .NET allows only 2 open HTTP connections to the same server so I immediately realized that it's not a coincidence (number 2 here and there).

Well, the HttpWebRequest does not have Dispose (nor Close). So there is nothing to close if WebException is thrown...

So i started to debug a hell out of it.
In debug I noticed that when I catch Exception it's actually WebException object and it has it's own Response stream.
Bingo... Nobody ever closes that stream..
George, it's possible that nobody ever needs to close that stream.

I've been doing some debugging in .NET 3.5. Recall that the source is available during debugging for .NET 3.5. I've been debugging into the following code:

HttpWebRequest httpWebRequest =

(HttpWebRequest) System.Net.WebRequest.Create("http://localhost/NEVEREVER");

HttpWebResponse httpWebResponse;

try

{

httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();

}

catch (WebException ex)

{

WebResponse resp = ex.Response;

((IDisposable) resp).Dispose();

}



What happens is that WebResponse.Dispose calls the virtual Close method, which is overridden by HttpWebResponse.Close. This calls Close on the stream. However, the stream was a System.Net.SyncMemoryStream, which does not overide Close. That is overridden by MemoryStream, which is a base class of SyncMemoryStream. Naturally, MemoryStream doesn't close any network connections. It just sets some flags and calls base.Dispose(disposing).



I looked at the WebResponse in the debugger, and noticed that the Headers collection contained a "Connection:close" header. I suggest you look at your WebResponse and see what's in it. It may already be closed, taking up no additional network resources. I set a breakpoint before the Dispose call, and issued a "netstat" from the command line. There were no connections open to localhost, so I think the connection was already closed.

 

[George Ter-Saakov]

Three points....

1. Who guarantees you that it's always going to be SyncMemoryStream...
I guess in some cases it is as in your test. In some cases it's not as it is not in my case. The behavior when I get two 404 exception and then could not connect to server is not something I could dream up easily .
Plus keep in mind everything started to work when I specifically started to catch WebException and closing WebException.Response stream.

2. what if Web server throws 500 error and produces 100 Meg stream with an explanation of the error. Will .NET dump all that into MemoryStream? Doubt that. Most likely it's conditional... Not sure based on what....

3. Microsoft developers fro .NET 3.5 do catch WebException and close WebException.Response object explicity. Most likely the guy who wrote it is aware that WebExcpetion.Response must be closed...

PS: I had that behavior on Windows Vista.. So if you have access to Vista you can try to run your test there and see if result will be different.

Thanks
George.



"John Saunders [MVP]" <john.saunders at trizetto.com> wrote in message I am positive...

I just run into it "face down".. I am writing a backup application that is backing up files/folders to S3 Amazon service.
And the way S3 works it's perfectly normal (with my application) to get 404 error which is end up being WebException

Then I noticed that after I get two 404 my application hangs and can not connect to Amazon service anymore (restarting application always fixed the problem).

I am well aware that by default .NET allows only 2 open HTTP connections to the same server so I immediately realized that it's not a coincidence (number 2 here and there).

Well, the HttpWebRequest does not have Dispose (nor Close). So there is nothing to close if WebException is thrown...

So i started to debug a hell out of it.
In debug I noticed that when I catch Exception it's actually WebException object and it has it's own Response stream.
Bingo... Nobody ever closes that stream..
George, it's possible that nobody ever needs to close that stream.

I've been doing some debugging in .NET 3.5. Recall that the source is available during debugging for .NET 3.5. I've been debugging into the following code:

HttpWebRequest httpWebRequest =

(HttpWebRequest) System.Net.WebRequest.Create("http://localhost/NEVEREVER");

HttpWebResponse httpWebResponse;

try

{

httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();

}

catch (WebException ex)

{

WebResponse resp = ex.Response;

((IDisposable) resp).Dispose();

}



What happens is that WebResponse.Dispose calls the virtual Close method, which is overridden by HttpWebResponse.Close. This calls Close on the stream. However, the stream was a System.Net.SyncMemoryStream, which does not overide Close. That is overridden by MemoryStream, which is a base class of SyncMemoryStream. Naturally, MemoryStream doesn't close any network connections. It just sets some flags and calls base.Dispose(disposing).



I looked at the WebResponse in the debugger, and noticed that the Headers collection contained a "Connection:close" header. I suggest you look at your WebResponse and see what's in it. It may already be closed, taking up no additional network resources. I set a breakpoint before the Dispose call, and issued a "netstat" from the command line. There were no connections open to localhost, so I think the connection was already closed.
--
--------------------------------------------------------------------------------
John Saunders | MVP - Windows Server System - Connected System Developer

 

[John Saunders [MVP]]

Three points....

1. Who guarantees you that it's always going to be SyncMemoryStream...
I guess in some cases it is as in your test. In some cases it's not as it is not in my case. The behavior when I get two 404 exception and then could not connect to server is not something I could dream up easily .
Plus keep in mind everything started to work when I specifically started to catch WebException and closing WebException.Response stream.

2. what if Web server throws 500 error and produces 100 Meg stream with an explanation of the error. Will .NET dump all that into MemoryStream? Doubt that. Most likely it's conditional... Not sure based on what....
I don't know. I only looked at the Dispose code, not the code that detects the 404 and throws the exception. I also don't know whether or not it will always use the same stream type.

In fact, I just tried the same code, but with a valid URL. In this case, the ResponseStream is of type System.Net.ConnectStream. Also, a "netstat" does show that the connection is open.

I think that you only get a connected stream when there has been no error. If there's an error, then I suspect that a copy of the response stream is made in the System.Net.SyncMemoryStream, so that it is no longer necessary to Dispose the Response.
3. Microsoft developers fro .NET 3.5 do catch WebException and close WebException.Response object explicity. Most likely the guy who wrote it is aware that WebExcpetion.Response must be closed...
I'm not sure what you mean. The closing I saw was only when Dispose was called on the Response.
PS: I had that behavior on Windows Vista.. So if you have access to Vista you can try to run your test there and see if result will be different.
If .NET behaved substantially differently on Vista, that would be a bug, in my opinion.

I'd suggest you create a Connect issue about the documentation of this. It should be clearly documented whether or not it is necessary to close ex.Response.