HttpWebResponse & Webservers

[Brent]

I'm having odd problems with the HttpWebResponse class. Some servers are
quite speedy, while others don't seem to want to talk to my code.
Consider the following pages*:

http://planetbrent.com/test.aspx?url=http://www.yahoo.com
http://planetbrent.com/test.aspx?url=http://www.msn.com
http://planetbrent.com/test.aspx?url=http://www.sec.gov
http://planetbrent.com/test.aspx?ur...5/000090266405001703/0000902664-05-001703.txt

The first two return a "screen scrape" quite rapidly -- all-but
instantly. The third and fourth links -- both on the same server, I'm
guessing -- take several seconds (if you test them, please be patient!).
The four pages are roughly the same size in bytes.

Viewing the slow pages in a normal browser from the same machine is
acceptably fast.

I'm thinking there may be an issue with the way the response comes back
from the server. Is it possible that some servers deliver data that's
confusing to the HttpWebResponse class?

I read something about bad chunking (or something like that) from Apache
servers, and that one possible solution was to get bytes instead of
lines. That might be a solution ... except that I need to read through
URLs like the fourth one line-by-line, as each line gets parsed by
other parts of the code.

I'd sure appreciate any pointers on handling this situation!

--Brent

*Full source code for the page.
=========================================================
<%@ Page Language="C#" %>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Web" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script language="C#" runat="server">
string rowsdeclared = "";

public void Page_Load(Object sender, EventArgs e)
{
string url = Request.QueryString["url"] != null ?
Request.QueryString["url"] : "http://www.yahoo.com";
ctrlSF.Text = "Started: " +
System.DateTime.Now.ToString("HH:mm:ss") + " | ";
ctrlText.Text = getHeader(url);
ctrlSF.Text += "Finished: " + System.DateTime.Now.ToString("HH:mm:ss");
}


public string getHeader (string strURL)
{
StringBuilder returnString = new StringBuilder();

String strreturn;
String thisRow = "";

WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();

using (StreamReader sr = new StreamReader(objResponse.GetResponseStream()))
{
while ((thisRow = sr.ReadLine()) != null)
{
returnString.Append(thisRow);
}
}
return returnString.ToString();
}

</script>

<asp:Literal id="ctrlSF" runat="server" />
<asp:Literal id="ctrlText" runat = "server" />

 

[Bruce Barker]

you need to run a network trace and determine where the delays are. for
example, win2000 has a tendency to use piggyback acks when it shouldn't and
can cause delays over http.

-- bruce (sqlwork.com)

I'm having odd problems with the HttpWebResponse class. Some servers are
quite speedy, while others don't seem to want to talk to my code. Consider
the following pages*:

http://planetbrent.com/test.aspx?url=http://www.yahoo.com
http://planetbrent.com/test.aspx?url=http://www.msn.com
http://planetbrent.com/test.aspx?url=http://www.sec.gov
http://planetbrent.com/test.aspx?ur...5/000090266405001703/0000902664-05-001703.txt

The first two return a "screen scrape" quite rapidly -- all-but instantly.
The third and fourth links -- both on the same server, I'm guessing --
take several seconds (if you test them, please be patient!). The four
pages are roughly the same size in bytes.

Viewing the slow pages in a normal browser from the same machine is
acceptably fast.

I'm thinking there may be an issue with the way the response comes back
from the server. Is it possible that some servers deliver data that's
confusing to the HttpWebResponse class?

I read something about bad chunking (or something like that) from Apache
servers, and that one possible solution was to get bytes instead of lines.
That might be a solution ... except that I need to read through URLs like
the fourth one line-by-line, as each line gets parsed by other parts of
the code.

I'd sure appreciate any pointers on handling this situation!

--Brent

*Full source code for the page.
=========================================================
<%@ Page Language="C#" %>
<%@ Import Namespace="System" %>
<%@ Import Namespace="System.Web" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.Text" %>
<%@ Import Namespace="System.Data" %>

<script language="C#" runat="server">
string rowsdeclared = "";

public void Page_Load(Object sender, EventArgs e)
{
string url = Request.QueryString["url"] != null ?
Request.QueryString["url"] : "http://www.yahoo.com";
ctrlSF.Text = "Started: " + System.DateTime.Now.ToString("HH:mm:ss") +
" | ";
ctrlText.Text = getHeader(url);
ctrlSF.Text += "Finished: " +
System.DateTime.Now.ToString("HH:mm:ss");
}


public string getHeader (string strURL)
{
StringBuilder returnString = new StringBuilder();

String strreturn;
String thisRow = "";

WebResponse objResponse;
WebRequest objRequest = HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();

using (StreamReader sr = new
StreamReader(objResponse.GetResponseStream()))
{
while ((thisRow = sr.ReadLine()) != null)
{
returnString.Append(thisRow);
}
}
return returnString.ToString();
}

</script>

<asp:Literal id="ctrlSF" runat="server" />
<asp:Literal id="ctrlText" runat = "server" />

 

[Brent]

Thanks, Bruce. I have Ethereal installed, but (frankly!) I don't have a
clue about all the bits of data it presents. There are a lot of
red-flagged lines, which I assume mean problems. Whatever. Funny thing:
two weeks ago this same code worked like a champ. Weird.

Is a network trace different from what Ethereal produces?

--Brent