Programmatic construction of httprequest?

B

Bill Cohagan

I'm constructing an ASP page that I'd like to test by writing a program that
simulates "many" users hitting the submit button on a form. I assume it's
possible to manually construct an httprequest object, but the docs are more
oriented towards using one that already exists. Am I on the right track or
is there an easier way?

In my case the Form to be submitted has a couple of hundred fields and what
I want to do is access a database with "answers" to use for filling in these
fields. I can then validate the results by comparing the source database
against what my ASP app creates.

Thanks for any suggestions/pointers...

Bill
 
G

Guest

The class that you want to look at is WebClient, specifically the
UploadValues method.

There may be automated tools that will simplify what you're trying to do.
 
J

Joerg Jooss

Bill said:
I'm constructing an ASP page that I'd like to test by writing a
program that simulates "many" users hitting the submit button on a
form. I assume it's possible to manually construct an httprequest
object, but the docs are more oriented towards using one that already
exists. Am I on the right track or is there an easier way?

In my case the Form to be submitted has a couple of hundred fields
and what I want to do is access a database with "answers" to use for
filling in these fields. I can then validate the results by comparing
the source database against what my ASP app creates.

Thanks for any suggestions/pointers...

System.Web.HttpRequest is a server-side class. To create your own HTTP
requests, you'll want to use System.Net.HttpWebRequest. The system you're
building reminds me of FIT --
see http://fit.c2.com/wiki.cgi?WhatsWhat.

Cheers,
 
S

Steven Cheng[MSFT]

Hi Bill,

As for using the HttpWebRequest, here ar e some tech articles :

#Test Automation for ASP.NET Web Apps with SSL
http://msdn.microsoft.com/msdnmag/issues/04/08/TestRun/default.aspx

#POSTing Data with ASP.NET
http://authors.aspalliance.com/stevesmith/articles/netscrape2.asp

Hope also helps.

Regards,

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

Get Preview at ASP.NET whidbey
http://msdn.microsoft.com/asp.net/whidbey/default.aspx
 
B

Bill Cohagan

Steven
I've constructed a WebRequest class in c# based on these articles, but am
experiencing an error (500) Internal Server Error. Following I'm including
the constructor and the Post method code:
====================
public WebRequest(string url)
{
req = (HttpWebRequest)System.Net.WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
}

public string Post()
{
req.ContentLength = post.Length;
using(StreamWriter myWriter = new StreamWriter(req.GetRequestStream()))
{
myWriter.Write(post);
}
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
using (StreamReader sr = new StreamReader(res.GetResponseStream()) )
{
return sr.ReadToEnd();
}
}
==============================================
The error is thrown on the call to req.GetResponse().

The url I'm passing in the constructor is http://localhost/QW/spike1.aspx
which is the correct url for the page under test. I've tried several
different values in the post string, including string.Empty, and always get
the same error. The _status property of the request indicates a protocol
error, but beyond that I'm not sure how to proceed. Any suggestions as to
how I might debug this?

Thanks
Bill
 
S

Scott Allen

Hi Bill:

In my experience, nine times out of ten this is caused by incorrect
POST data. You might need to UrlEncode the data, or might not have the
right content length, something along those lines.

One of the best ways to debug this is to use a tool like Fiddler.
http://www.fiddlertool.com/fiddler/

What you do is record what your browser sends to the server if you do
the POST manually. Then record what your sofrware sends. Then compare
the two and see what is different.

I have an article here that may offer some insight too:

Screen Scraping, ViewState, and Authentication using ASP.Net
http://odetocode.com/Articles/162.aspx

Hope this helps,
 
B

Bill Cohagan

Scott-
Thanks for the response and suggestions. I downloaded and installed
fiddler and it works fine for a browser induced POST, but absolutely nothing
is captured when I try my program. I have included the suggested
GlobalProxySelection.Select code as well.

Any ideas/suggestions?

Thanks again,
Bill
 
S

Scott Allen

Hi Bill:

I'm sorry I don't have a good suggestion (I've seen it work,
really!!).

Perhaps restarting IIS after Fiddler has started? Seems hackish but I
can't think of anything at the moment....
 
J

Joerg Jooss

Bill said:
Steven
I've constructed a WebRequest class in c# based on these articles,
but am experiencing an error (500) Internal Server Error. Following
I'm including the constructor and the Post method code:
====================
public WebRequest(string url)
{
req = (HttpWebRequest)System.Net.WebRequest.Create(url);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
}

public string Post()
{
req.ContentLength = post.Length;


First law of proper character string processing: String length in general
does not equal byte length.

Because...

using(StreamWriter myWriter = new
StreamWriter(req.GetRequestStream())) {
myWriter.Write(post);

.... posts UTF-8 encoded text. Every non-US-ASCII characters will take two or
more bytes to encode -- although admittedly at least some of your tests
should have worked.
The url I'm passing in the constructor is
http://localhost/QW/spike1.aspx which is the correct url for the page
under test. I've tried several different values in the post string,
including string.Empty, and always get the same error. The _status
property of the request indicates a protocol error, but beyond that
I'm not sure how to proceed. Any suggestions as to how I might debug
this?

As Steve suggested, try to get Fiddler to work. Or try this sample:

public void PostForm(string url, string formData, string encoding) {
byte[] content = Encoding.GetEncoding(encoding).GetByte(formData);
string contentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encoding);

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = contentType;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream()) {
requestStream.Write(content, 0, content.Length);
}

HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream()) {
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Do something useful with buffer
}
}

Cheers,
 
B

Bill Cohagan

Joerg
I tried the code you supplied, but I'm getting the same error. More
exactly here's the code I used:
===================
public void PostForm(string url, string formData, string encoding)
{
byte[] content = Encoding.GetEncoding(encoding).GetBytes(formData);
string contentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encoding);
HttpWebRequest request = (HttpWebRequest) System.Net.WebRequest.Create(url);
request.Method = "POST";
request.ContentType = contentType;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(content, 0, content.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Do something useful with buffer
}
}
=================
The code that calls your method is;

PostForm("http://localhost/QW/spike1.aspx", "foo=bar", "utf-8");

I get the same error(The remote server returned an error: (500) Internal
Server Error.) at the line that calls GetResponse().

Any suggestions?

Thanks in advance,
Bill
 
J

Joerg Jooss

Bill said:
Joerg
I tried the code you supplied, but I'm getting the same error. More
exactly here's the code I used:
===================
public void PostForm(string url, string formData, string encoding)
{
byte[] content = Encoding.GetEncoding(encoding).GetBytes(formData);
string contentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encoding);
HttpWebRequest request = (HttpWebRequest)
System.Net.WebRequest.Create(url); request.Method = "POST";
request.ContentType = contentType;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(content, 0, content.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Do something useful with buffer
}
}
=================
The code that calls your method is;

PostForm("http://localhost/QW/spike1.aspx", "foo=bar", "utf-8");

I get the same error(The remote server returned an error: (500)
Internal Server Error.) at the line that calls GetResponse().

Any suggestions?

Oops, it seems I missed the fact that you get a 500 -- because in this case,
it's quite likely that this is a server-side problem. I guess you can debug
the ASP.NET page? Does the request pass through the ASP.NET HTTP pipeline or
is rejected immediately?

Cheers,
 
B

Bill Cohagan

Joerg
I'm not sure if it passes throught the "pipeline", but it doesn't get far
enough to show up in Fiddler. The spike1.aspx page itself seems to work OK
if I access it via the browser; i.e., I actually submit a form. Also, if I
set a breakpoint in the page load event handler for spike1 it never fires.
So my guess is that the error occurs before my asp page gets loaded. At one
point I found a property that indicated it was a protocol error, but I don't
seem to be able to find that now (when the program breaks on the error.)
Since it appears not to make it to the asp page I don't really know how to
debug this.

I've checked the logs, but don't find anything useful there either. Also,
I've tried this on a couple of machines and get the same error -- so if it's
a config problem it's common to both installations.

Thanks for the help.

Bill

Joerg Jooss said:
Bill said:
Joerg
I tried the code you supplied, but I'm getting the same error. More
exactly here's the code I used:
===================
public void PostForm(string url, string formData, string encoding)
{
byte[] content = Encoding.GetEncoding(encoding).GetBytes(formData);
string contentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encoding);
HttpWebRequest request = (HttpWebRequest)
System.Net.WebRequest.Create(url); request.Method = "POST";
request.ContentType = contentType;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(content, 0, content.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Do something useful with buffer
}
}
=================
The code that calls your method is;

PostForm("http://localhost/QW/spike1.aspx", "foo=bar", "utf-8");

I get the same error(The remote server returned an error: (500)
Internal Server Error.) at the line that calls GetResponse().

Any suggestions?

Oops, it seems I missed the fact that you get a 500 -- because in this case,
it's quite likely that this is a server-side problem. I guess you can debug
the ASP.NET page? Does the request pass through the ASP.NET HTTP pipeline or
is rejected immediately?

Cheers,
 
J

Joerg Jooss

Bill said:
I'm not sure if it passes throught the "pipeline", but it doesn't
get far enough to show up in Fiddler.

Oops. So that means this setup

ClientApp --> Fiddler --> WebApp

produces no output in Fiddler? Did you change the code so it uses the proxy?

The spike1.aspx page itself
seems to work OK if I access it via the browser; i.e., I actually
submit a form. Also, if I set a breakpoint in the page load event
handler for spike1 it never fires. So my guess is that the error
occurs before my asp page gets loaded. At one point I found a
property that indicated it was a protocol error, but I don't seem to
be able to find that now (when the program breaks on the error.)
Since it appears not to make it to the asp page I don't really know
how to debug this.

I've checked the logs, but don't find anything useful there either.
Also, I've tried this on a couple of machines and get the same error
-- so if it's a config problem it's common to both installations.

What's this spike.aspx doing anyway? Can you substitute it for testing
purposes with a another page -- assuming it is not a test page already?

Cheers,
 
B

Bill Cohagan

Joerg
Yes, I added the code to explicitly enable the proxy, but still no Fiddler
entries at all. Spike1.aspx is intended to take the data from the form and
stuff it into a database, but of course I could substitute a do nothing
page. I haven't done so because it appears that we're never getting to the
page in the first place. I suppose it's possible that the page is the source
of the error, but that implies:

1.) My breakpoint on the page load handler isn't working.
2.) Fidler isn't catching the request.

I suppose that it wouldn't hurt to try however. I'll let you know what
happens. Let me know if you think of any other ideas on trouble shooting
this. Also, if you have the time I'd appreciate it if you'd try out your
code on your machine just as a sanity check.

Thanks again for the help.

Bill
 
J

Joerg Jooss

Bill said:
Yes, I added the code to explicitly enable the proxy, but still no
Fiddler entries at all. Spike1.aspx is intended to take the data from
the form and stuff it into a database, but of course I could
substitute a do nothing page. I haven't done so because it appears
that we're never getting to the page in the first place. I suppose
it's possible that the page is the source of the error, but that
implies:

1.) My breakpoint on the page load handler isn't working.
2.) Fidler isn't catching the request.

But this quite dodgy. You get a 500 on the serverside, but you don't see the
request in Fiddler...
I suppose that it wouldn't hurt to try however. I'll let you know
what happens. Let me know if you think of any other ideas on trouble
shooting this.
Also, if you have the time I'd appreciate it if you'd
try out your code on your machine just as a sanity check.

Yes, it works. I've posted this code numerous times in various .NET groups
as sample code.

Cheers,
 
B

Bill Cohagan

Joerg
I assumed that you hadn't run this code since it contains an error that
won't let it compile; i.e., GetByte should be GetBytes in the expression
defining the content variable. ;-)

Bill
 
B

Bill Cohagan

Joerg
It appears that the 500 error is masking an error being thrown in my page
load event handler -- so obviously I've missed something about breakpoints
in ASP code. I discovered this by commenting out the worker code in that
method and rerunning the test. The GetResponse call now returns without
throwing an error. I have not yet determined the nature of the actual error,
but pretty clearly it's my own code.

I'm still confused by the absence of Fiddler output and by the breakpoint
not working. Once I fix the underlying code problem I'll explore that
further.

Thanks again for the help.
Bill
 
J

Joerg Jooss

Bill said:
Joerg
I assumed that you hadn't run this code since it contains an error
that won't let it compile; i.e., GetByte should be GetBytes in the
expression defining the content variable. ;-)

Ouch, you're right... I just grabbed the code from one of my earlier
posts... I'm sorry. BTW, here's working sample with URL-encoding:

public void PostForm(string url, string formData, string encodingName) {
MemoryStream content = new MemoryStream(formData.Length * 2);
Encoding encoding = Encoding.GetEncoding(encodingName);
string[] keyValuePairs = formData.Split('&', '=');
int numberOfPairs = keyValuePairs.Length;
bool isKey = true;
foreach (string keyValue in keyValuePairs) {
byte[] bytes = HttpUtility.UrlEncodeToBytes(keyValue, encoding);
content.Write(bytes, 0, bytes.Length);
if (isKey) {
content.WriteByte((byte) '=');
}
else {
content.WriteByte((byte) '&');
}
isKey = !isKey;
}
content.SetLength(content.Length - 1);

HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encodingName);;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream()) {
content.WriteTo(requestStream);
}

HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream()) {
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// use buffer...
}
}

Cheers,
 
B

Bill Cohagan

Joerg
I've determined the cause of my breakpoints not working. I wasn't aware of
the necessity to attach to the asp process in order for the breakpoints to
work. Once I figured that out then I was able to do some stepping and figure
out the "real" problem. It's still a mystery why Fiddle wasn't seeing
anything. It's quite possible I'm doing something wrong there as well -- so
I'll probably push on that a bit and see if I can figure it out.

Thanks again for sticking with me on this problem.

Bill

Bill Cohagan said:
Joerg
I'm not sure if it passes throught the "pipeline", but it doesn't get far
enough to show up in Fiddler. The spike1.aspx page itself seems to work OK
if I access it via the browser; i.e., I actually submit a form. Also, if I
set a breakpoint in the page load event handler for spike1 it never fires.
So my guess is that the error occurs before my asp page gets loaded. At one
point I found a property that indicated it was a protocol error, but I don't
seem to be able to find that now (when the program breaks on the error.)
Since it appears not to make it to the asp page I don't really know how to
debug this.

I've checked the logs, but don't find anything useful there either. Also,
I've tried this on a couple of machines and get the same error -- so if it's
a config problem it's common to both installations.

Thanks for the help.

Bill

Joerg Jooss said:
Bill said:
Joerg
I tried the code you supplied, but I'm getting the same error. More
exactly here's the code I used:
===================
public void PostForm(string url, string formData, string encoding)
{
byte[] content = Encoding.GetEncoding(encoding).GetBytes(formData);
string contentType = String.Format(
"application/x-www-form-urlencoded; charset={0}", encoding);
HttpWebRequest request = (HttpWebRequest)
System.Net.WebRequest.Create(url); request.Method = "POST";
request.ContentType = contentType;
request.ContentLength = content.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(content, 0, content.Length);
}
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
byte[] buffer = new byte[response.ContentLength];
responseStream.Read(buffer, 0, buffer.Length);
// Do something useful with buffer
}
}
=================
The code that calls your method is;

PostForm("http://localhost/QW/spike1.aspx", "foo=bar", "utf-8");

I get the same error(The remote server returned an error: (500)
Internal Server Error.) at the line that calls GetResponse().

Any suggestions?

Oops, it seems I missed the fact that you get a 500 -- because in this case,
it's quite likely that this is a server-side problem. I guess you can debug
the ASP.NET page? Does the request pass through the ASP.NET HTTP
pipeline
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top