submit form via HttpWebRequest or WebClient

Discussion in 'ASP .Net' started by John A Grandy, Aug 24, 2004.

  1. has anyone successfully used HttpWebRequest or WebClient class to simulate
    submission of a simple HTML form?

    for example: a very simple plain-vanilla form with a textbox and a button.
    when the button is clicked the form is submitted with the textbox contents.

    could you please post some sample code? thanks.
    John A Grandy, Aug 24, 2004
    #1
    1. Advertising

  2. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > has anyone successfully used HttpWebRequest or WebClient class to
    > simulate submission of a simple HTML form?
    >
    > for example: a very simple plain-vanilla form with a textbox and a
    > button. when the button is clicked the form is submitted with the
    > textbox contents.
    >
    > could you please post some sample code? thanks.


    That's a full blown WebRequest/WebResponse sample including URL encoding.


    public void PostForm(string url, string formData, string encodingName) {
    // URL encode the post data, applying the character encoding given by
    // encodingName.
    MemoryStream content = new MemoryStream(formData.Length * 2);
    Encoding encoding = Encoding.GetEncoding(encodingName);
    string[] keyValuePairs = formData.Split('&', '=');
    int numberOfPairs = keyValuePairs.Length;
    bool isKey = true;
    foreach (string keyValue in keyValuePairs) {
    byte[] bytes = HttpUtility.UrlEncodeToBytes(keyValue, encoding);
    content.Write(bytes, 0, bytes.Length);
    if (isKey) {
    content.WriteByte((byte) '=');
    }
    else {
    content.WriteByte((byte) '&');
    }
    isKey = !isKey;
    }
    // We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
    anyway
    content.SetLength(content.Length - 1);

    // That's the actual HTTP part of the sample
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = String.Format(
    "application/x-www-form-urlencoded; charset={0}", encodingName);
    request.ContentLength = content.Length;
    using (Stream requestStream = request.GetRequestStream()) {
    content.WriteTo(requestStream);
    }

    HttpWebResponse response = (HttpWebResponse) request.GetResponse();
    using (Stream responseStream = response.GetResponseStream()) {
    byte[] buffer = new byte[response.ContentLength];
    responseStream.Read(buffer, 0, buffer.Length);
    // Process buffer... e.g. write to file, decode to string, etc.
    }
    }

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 24, 2004
    #2
    1. Advertising

  3. John A Grandy

    Scott Allen Guest

    Hi John:

    I have an example here:

    Screen Scraping, ViewState, and Authentication using ASP.Net
    http://odetocode.com/Articles/162.aspx

    This also shows how to extract the viewstate out of an asp.net page to
    post back.

    HTH,

    --
    Scott
    http://www.OdeToCode.com

    On Mon, 23 Aug 2004 23:23:31 -0700, "John A Grandy"
    <johnagrandy-at-yahoo.com> wrote:

    >has anyone successfully used HttpWebRequest or WebClient class to simulate
    >submission of a simple HTML form?
    >
    >for example: a very simple plain-vanilla form with a textbox and a button.
    >when the button is clicked the form is submitted with the textbox contents.
    >
    >could you please post some sample code? thanks.
    >
    Scott Allen, Aug 24, 2004
    #3
  4. so whatever data the user would enter into the text box "textbox1", i
    simulate by composing the string

    formData="textbox1=thedata"

    but how do i know what encodingName should be ?


    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > has anyone successfully used HttpWebRequest or WebClient class to
    > > simulate submission of a simple HTML form?
    > >
    > > for example: a very simple plain-vanilla form with a textbox and a
    > > button. when the button is clicked the form is submitted with the
    > > textbox contents.
    > >
    > > could you please post some sample code? thanks.

    >
    > That's a full blown WebRequest/WebResponse sample including URL encoding.
    >
    >
    > public void PostForm(string url, string formData, string encodingName) {
    > // URL encode the post data, applying the character encoding given by
    > // encodingName.
    > MemoryStream content = new MemoryStream(formData.Length * 2);
    > Encoding encoding = Encoding.GetEncoding(encodingName);
    > string[] keyValuePairs = formData.Split('&', '=');
    > int numberOfPairs = keyValuePairs.Length;
    > bool isKey = true;
    > foreach (string keyValue in keyValuePairs) {
    > byte[] bytes = HttpUtility.UrlEncodeToBytes(keyValue, encoding);
    > content.Write(bytes, 0, bytes.Length);
    > if (isKey) {
    > content.WriteByte((byte) '=');
    > }
    > else {
    > content.WriteByte((byte) '&');
    > }
    > isKey = !isKey;
    > }
    > // We'll end up with one surplus '&' -- shouldn't hurt, but we cut it
    > anyway
    > content.SetLength(content.Length - 1);
    >
    > // That's the actual HTTP part of the sample
    > HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    > request.Method = "POST";
    > request.ContentType = String.Format(
    > "application/x-www-form-urlencoded; charset={0}", encodingName);
    > request.ContentLength = content.Length;
    > using (Stream requestStream = request.GetRequestStream()) {
    > content.WriteTo(requestStream);
    > }
    >
    > HttpWebResponse response = (HttpWebResponse) request.GetResponse();
    > using (Stream responseStream = response.GetResponseStream()) {
    > byte[] buffer = new byte[response.ContentLength];
    > responseStream.Read(buffer, 0, buffer.Length);
    > // Process buffer... e.g. write to file, decode to string, etc.
    > }
    > }
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 24, 2004
    #4
  5. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > so whatever data the user would enter into the text box "textbox1", i
    > simulate by composing the string
    >
    > formData="textbox1=thedata"


    Right.

    > but how do i know what encodingName should be ?


    That's the tricky part, because this depends on the web application. But
    there's a simple heuristic that should work for most sites. Use the same
    encoding that the site uses as character encoding. Open the web page and
    check out what character encoding your browser applies (in IE
    View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western Europe
    (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->

    Unless you're working with a non US or non Western European web site, those
    are the ones you'll most likely encounter. In case of doubt, use ISO-8859-1.

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 24, 2004
    #5
  6. very interesting ....

    unfortunately, i'm in a stickier situation than communicating with a
    public-facing website.

    i'm communicating over vpn with a private web-server meant for
    data-provision to intermediaries.

    the web-server does not validate credentials. it just immediately returns
    data (interesting, as an xml doc)

    the only working tool i have is a java applet written by someone else
    (someone who is long gone).

    the java applet is very simple. just a textbox and a submit button.

    a long alphanumeric string is entered into the textbox (it ends with "^").

    clicking the button fires a js function that calls form.submit and the form
    is posted.

    "view source" on the IE browseer hosting the java applet reveals only a very
    simple block of html

    a <textarea> element is used for data-entry.

    i can't disclose the actual url -- but its similar to
    "https://www.somecompany.com:5555/someservice.asmx/somefunction"

    .... so apparently the web-server is hosting a .net web-service.


    i tried your method. i constructed using the data-string successfully used
    in the Java applet :

    postData = "myTextArea=dataString";

    encodingName = "utf-8";

    request.GetResponse();

    triggered a runtime error : "500 : internal server error"


    do you think the problem might be with encodingName? what do you think
    good possibilities are ?

    or could the problem be in some other area ?





    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > so whatever data the user would enter into the text box "textbox1", i
    > > simulate by composing the string
    > >
    > > formData="textbox1=thedata"

    >
    > Right.
    >
    > > but how do i know what encodingName should be ?

    >
    > That's the tricky part, because this depends on the web application. But
    > there's a simple heuristic that should work for most sites. Use the same
    > encoding that the site uses as character encoding. Open the web page and
    > check out what character encoding your browser applies (in IE
    > View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

    Europe
    > (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->
    >
    > Unless you're working with a non US or non Western European web site,

    those
    > are the ones you'll most likely encounter. In case of doubt, use

    ISO-8859-1.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 25, 2004
    #6
  7. also, the MSDN docs for Encoding.GetEncoding(name) do not list "ISO-8859-1"



    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > so whatever data the user would enter into the text box "textbox1", i
    > > simulate by composing the string
    > >
    > > formData="textbox1=thedata"

    >
    > Right.
    >
    > > but how do i know what encodingName should be ?

    >
    > That's the tricky part, because this depends on the web application. But
    > there's a simple heuristic that should work for most sites. Use the same
    > encoding that the site uses as character encoding. Open the web page and
    > check out what character encoding your browser applies (in IE
    > View->Encoding). Western Europe (ISO) is ISO-8859-1, whereas Western

    Europe
    > (Windows) is Windows-1252. Unicoded (UTF-8) is, um, UTF-8 ;->
    >
    > Unless you're working with a non US or non Western European web site,

    those
    > are the ones you'll most likely encounter. In case of doubt, use

    ISO-8859-1.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 25, 2004
    #7
  8. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > also, the MSDN docs for Encoding.GetEncoding(name) do not list
    > "ISO-8859-1"


    They just show a few names in the .NET docs. The list of all supported
    encodings is hidden deeply within MSDN, and of course it's at some place
    you'd never expect it to be ;-)

    Start here:

    http://www.microsoft.com/globaldev/getWR/steps/wrg_codepage.mspx

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 25, 2004
    #8
  9. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > very interesting ....
    >
    > unfortunately, i'm in a stickier situation than communicating with a
    > public-facing website.

    [...]
    > i can't disclose the actual url -- but its similar to
    > "https://www.somecompany.com:5555/someservice.asmx/somefunction"
    >
    > ... so apparently the web-server is hosting a .net web-service.


    Yes.

    > i tried your method. i constructed using the data-string
    > successfully used in the Java applet :
    >
    > postData = "myTextArea=dataString";
    >
    > encodingName = "utf-8";
    >
    > request.GetResponse();
    >
    > triggered a runtime error : "500 : internal server error"
    >
    >
    > do you think the problem might be with encodingName? what do you
    > think good possibilities are ?
    >
    > or could the problem be in some other area ?


    I guess the whole approrach could be wrong then. My sample code showed a
    programmatic web form submission, but you probably need a SOAP client.
    Actually, if it is a "proper" Web Service, VS .NET should be able to
    generate the necessary client-side code for you. The test code you've posted
    doesn't fit into the picture at all, though.

    I guess you'll need to talk to the Web Service guys to find out what
    requests their web service can handle.

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 25, 2004
    #9
  10. Hi Joerg ....

    I don't think the approach you suggested is wrong.

    I have a simple HTML page that submits its form to the web-server, and
    successfully receives a response. The HTML form odes not communicate in any
    way with any entity other than the following:

    The JavaScript function triggered by the button click is very very simple :
    all is does is a form.submit. There is no other client-side functionality
    occurring.

    Whatever method this company is using to process information on their side ,
    they do accept posting of a simple form with

    "InputData=0123456789abcdefghijklmnopqrstuvwxyz^" (for example)

    Is there any problem with URL encoding the "^" character?

    I think the encoding-type is probably the villian. I'll have to try various
    possibilities.

    Also, do you know various common headers that could be added? I see in
    many of the postings to this newsgroup that quite a different number of
    headers are being used.



    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > very interesting ....
    > >
    > > unfortunately, i'm in a stickier situation than communicating with a
    > > public-facing website.

    > [...]
    > > i can't disclose the actual url -- but its similar to
    > > "https://www.somecompany.com:5555/someservice.asmx/somefunction"
    > >
    > > ... so apparently the web-server is hosting a .net web-service.

    >
    > Yes.
    >
    > > i tried your method. i constructed using the data-string
    > > successfully used in the Java applet :
    > >
    > > postData = "myTextArea=dataString";
    > >
    > > encodingName = "utf-8";
    > >
    > > request.GetResponse();
    > >
    > > triggered a runtime error : "500 : internal server error"
    > >
    > >
    > > do you think the problem might be with encodingName? what do you
    > > think good possibilities are ?
    > >
    > > or could the problem be in some other area ?

    >
    > I guess the whole approrach could be wrong then. My sample code showed a
    > programmatic web form submission, but you probably need a SOAP client.
    > Actually, if it is a "proper" Web Service, VS .NET should be able to
    > generate the necessary client-side code for you. The test code you've

    posted
    > doesn't fit into the picture at all, though.
    >
    > I guess you'll need to talk to the Web Service guys to find out what
    > requests their web service can handle.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 25, 2004
    #10
  11. "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > very interesting ....
    > >
    > > unfortunately, i'm in a stickier situation than communicating with a
    > > public-facing website.

    > [...]
    > > i can't disclose the actual url -- but its similar to
    > > "https://www.somecompany.com:5555/someservice.asmx/somefunction"
    > >
    > > ... so apparently the web-server is hosting a .net web-service.

    >
    > Yes.
    >
    > > i tried your method. i constructed using the data-string
    > > successfully used in the Java applet :
    > >
    > > postData = "myTextArea=dataString";
    > >
    > > encodingName = "utf-8";
    > >
    > > request.GetResponse();
    > >
    > > triggered a runtime error : "500 : internal server error"
    > >
    > >
    > > do you think the problem might be with encodingName? what do you
    > > think good possibilities are ?
    > >
    > > or could the problem be in some other area ?

    >
    > I guess the whole approrach could be wrong then. My sample code showed a
    > programmatic web form submission, but you probably need a SOAP client.
    > Actually, if it is a "proper" Web Service, VS .NET should be able to
    > generate the necessary client-side code for you. The test code you've

    posted
    > doesn't fit into the picture at all, though.
    >
    > I guess you'll need to talk to the Web Service guys to find out what
    > requests their web service can handle.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 26, 2004
    #11
  12. are the web.config <webServices> settings related to successfully posting to
    a remote web-service ?

    or do these settings only relate to the hosting of web-services of
    web-servers ... ?


    for example, by default , the HttpPost setting is not present in web.config
    .....



    web.config file

    <webServices>

    <protocols>

    <add name="HttpPost" />

    </protocols>

    </webServices>





    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > very interesting ....
    > >
    > > unfortunately, i'm in a stickier situation than communicating with a
    > > public-facing website.

    > [...]
    > > i can't disclose the actual url -- but its similar to
    > > "https://www.somecompany.com:5555/someservice.asmx/somefunction"
    > >
    > > ... so apparently the web-server is hosting a .net web-service.

    >
    > Yes.
    >
    > > i tried your method. i constructed using the data-string
    > > successfully used in the Java applet :
    > >
    > > postData = "myTextArea=dataString";
    > >
    > > encodingName = "utf-8";
    > >
    > > request.GetResponse();
    > >
    > > triggered a runtime error : "500 : internal server error"
    > >
    > >
    > > do you think the problem might be with encodingName? what do you
    > > think good possibilities are ?
    > >
    > > or could the problem be in some other area ?

    >
    > I guess the whole approrach could be wrong then. My sample code showed a
    > programmatic web form submission, but you probably need a SOAP client.
    > Actually, if it is a "proper" Web Service, VS .NET should be able to
    > generate the necessary client-side code for you. The test code you've

    posted
    > doesn't fit into the picture at all, though.
    >
    > I guess you'll need to talk to the Web Service guys to find out what
    > requests their web service can handle.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 26, 2004
    #12
  13. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > are the web.config <webServices> settings related to successfully
    > posting to a remote web-service ?
    >
    > or do these settings only relate to the hosting of web-services of
    > web-servers ... ?
    >
    >
    > for example, by default , the HttpPost setting is not present in
    > web.config ....
    >
    >
    >
    > web.config file
    >
    > <webServices>
    >
    > <protocols>
    >
    > <add name="HttpPost" />
    >
    > </protocols>
    >
    > </webServices>


    You're right, these lines control whether an ASP.NET based web service
    accepts "pure" (non SOAP) requests. The line <add name="httPost"/> should
    enable posting. Note this is *disabled* in ASP.NET 1.1 by default.

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 26, 2004
    #13
  14. so, for my purposes of communicating with a remote web service (hosted by a
    commercial company) ...

    it is *their* responsibility to make sure that their web.config (or
    machine.config) is properly setup to accept HTTPPost requests ... ?


    "Joerg Jooss" <> wrote in message
    news:...
    > John A Grandy wrote:
    > > are the web.config <webServices> settings related to successfully
    > > posting to a remote web-service ?
    > >
    > > or do these settings only relate to the hosting of web-services of
    > > web-servers ... ?
    > >
    > >
    > > for example, by default , the HttpPost setting is not present in
    > > web.config ....
    > >
    > >
    > >
    > > web.config file
    > >
    > > <webServices>
    > >
    > > <protocols>
    > >
    > > <add name="HttpPost" />
    > >
    > > </protocols>
    > >
    > > </webServices>

    >
    > You're right, these lines control whether an ASP.NET based web service
    > accepts "pure" (non SOAP) requests. The line <add name="httPost"/> should
    > enable posting. Note this is *disabled* in ASP.NET 1.1 by default.
    >
    > Cheers,
    >
    > --
    > Joerg Jooss
    >
    >
    >
    John A Grandy, Aug 26, 2004
    #14
  15. John A Grandy

    Joerg Jooss Guest

    John A Grandy wrote:
    > so, for my purposes of communicating with a remote web service
    > (hosted by a commercial company) ...
    >
    > it is *their* responsibility to make sure that their web.config (or
    > machine.config) is properly setup to accept HTTPPost requests ... ?


    *Technically* yes.

    But in the end, this is decision between business partners ("We want POST!"
    "We have SOAP!") and of course of IT security. If I was concerned: They
    offer the service, they make they rules, and they need to tell you what you
    can do and what you should do.

    Cheers,

    --
    Joerg Jooss
    Joerg Jooss, Aug 26, 2004
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jeff Baker

    WebClient form Posting with form name?

    Jeff Baker, Aug 28, 2003, in forum: ASP .Net
    Replies:
    2
    Views:
    735
    John Saunders
    Aug 29, 2003
  2. Lars Netzel
    Replies:
    3
    Views:
    5,132
    Prakash
    Nov 7, 2005
  3. Natalia
    Replies:
    4
    Views:
    759
    Natalia
    Dec 5, 2006
  4. Tony Hunter

    Form Post via HttpWebRequest

    Tony Hunter, Mar 10, 2005, in forum: ASP .Net Security
    Replies:
    2
    Views:
    200
    alexdp
    Mar 16, 2005
  5. Replies:
    3
    Views:
    654
Loading...

Share This Page