Using HttpWebRequest to post form data with cookies

  • Thread starter Luis Esteban Valencia Muñoz
  • Start date
L

Luis Esteban Valencia Muñoz

I'm working on a web scraping application that needs to log into a website
before it can get the data to scrape. I've always been confused about how
the HttpWebRequest and HttpWebResponse objects work together with cookies,
and was hoping that someone here could clear it up for me!

Here's the steps I need to accomplish:

1) Set two cookies containing information I already have
2) Request the login page URL, and save a third cookie that is given to me
from that page
3) Submit a form via POST, along with the 3 cookies
4) Save a fourth cookie that is set after the form is submitted with valid
information

I don't need the entire code to do this, but rather some help with the order
to do it. Do I create a request and THEN set the cookies? Can I use the
same CookieContainer for each request? If someone could point me in the
right direction it would be greatly appreciated!
 
J

Joerg Jooss

Luis said:
I'm working on a web scraping application that needs to log into a
website before it can get the data to scrape. I've always been
confused about how the HttpWebRequest and HttpWebResponse objects
work together with cookies, and was hoping that someone here could
clear it up for me!

Here's the steps I need to accomplish:

1) Set two cookies containing information I already have
2) Request the login page URL, and save a third cookie that is given
to me from that page
3) Submit a form via POST, along with the 3 cookies
4) Save a fourth cookie that is set after the form is submitted with
valid information

I don't need the entire code to do this, but rather some help with
the order to do it. Do I create a request and THEN set the cookies?
Can I use the same CookieContainer for each request? If someone
could point me in the right direction it would be greatly appreciated!

This topic has been beaten to death by now -- just search for
WebRequest and CookieContainer on Google: http://tinyurl.com/9yp6r

Just make sure to use one CookieContainer instance throughout your web
conversation. It will pick up all cookies, and you don't need to copy
cookies from responses back to it -- the CookieContainer picks them up
automatically.

You can set any request header (including cookies) as long as you
haven't written to the request stream or read from the response stream
yet.

Cheers,
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top