Strange problem with HttpWebRequest

Discussion in 'ASP .Net' started by Blackstar, Nov 29, 2006.

  1. Blackstar

    Blackstar Guest

    I have a website I am trying to download a file from. This website is
    strange in the fact that there is no direct link to the URL of the
    file. The way the site works is as follows:

    1) Enter in search criteria
    2) Click on Download button to go to download page
    3) Click on Download button on Download page
    4) Server sends 2 bits of data to a script. The data sent does not
    include the query data. I'm assuming that the data is stored either in
    a server side session variable, or server-side on a temporary database.
    5) The script processes and initiates a file download. Inside any of
    the web pages, there are NO direct links directly to a file to
    download. Once the file is compiled, the webste starts streaming the
    data to the standard IE file download dialog. I need to capture this
    file that is downloaded into a stream of some sort, then save it out to
    a text file. The exported data is a text file.

    Here is the code I am trying to use:

    <------ CODE -------->
    Dim fileURL As New Uri("http://www.somesite.com")
    Dim myFileWebRequest As WebRequest =
    WebRequest.Create(fileURL)
    myFileWebRequest.Method = WebRequestMethods.Http.Post
    Dim buffer As Byte() =
    Encoding.UTF8.GetBytes("type=normal&cmd=DOWNLOAD")

    myFileWebRequest.ContentType =
    "application/x-www-form-urlencoded"
    myFileWebRequest.ContentLength = buffer.Length
    myFileWebRequest.Headers.Add(HttpRequestHeader.Cookie,
    wb.Document.Cookie)
    'myFileWebRequest.Headers.Add(HttpRequestHeader.Referer,
    "http://www.somesite.com/resource/servlet/search")
    'myFileWebRequest.Headers.Add(HttpRequestHeader.Accept,
    "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
    application/x-shockwave-flash, application/vnd.ms-excel,
    application/vnd.ms-powerpoint, application/msword, */*")

    myFileWebRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us")

    myFileWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,
    deflate")
    'myFileWebRequest.Headers.Add(HttpRequestHeader.UserAgent,
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
    1.1.4322; .NET CLR 2.0.50727)")
    'myFileWebRequest.Headers.Add(HttpRequestHeader.Host,
    "www.somesite.com")
    'myFileWebRequest.Headers.Add(HttpRequestHeader.Connection,
    "Keep-Alive")

    myFileWebRequest.Headers.Add(HttpRequestHeader.CacheControl,
    "no-cache")

    Dim myFileWebResponse As WebResponse =
    CType(myFileWebRequest.GetResponse, HttpWebResponse)
    Dim receivestream As Stream =
    myFileWebResponse.GetResponseStream
    Dim encode As Encoding =
    System.Text.Encoding.GetEncoding("utf-8")
    Dim readStream As New StreamReader(receivestream, encode)
    Dim read(256) As Char
    Dim count As Integer = readStream.Read(read, 0, 256)
    Dim objWriter As New StreamWriter(sFileName)

    While count > 0
    Dim str As New String(read, 0, count)

    objWriter.Write(str)
    count = readStream.Read(read, 0, 256)
    End While
    readStream.Close()
    objWriter.Close()

    myFileWebResponse.Close()

    <--------- END CODE --------->

    I am not getting ANY errors or exceptions, the program just hangs on
    the following line:
    Dim myFileWebResponse As WebResponse =
    CType(myFileWebRequest.GetResponse, HttpWebResponse)

    It will sit on this line forever if you let it. The other problem is
    that the website denies GET requests. It only allows POST requests.
    I've had a number of other programmers look at it, and they are just as
    stumped as me. If anyone can help, it would be most appreciated!

    Also, if you look in the above code, you will notice a large number of
    lines commented out that are designed to set items in the header. If I
    uncomment them, then the program hangs and tells me that I am not
    assigning it correctly and I need to use the name parameter. If the
    other header items that are uncomment work, then shouldn't the
    commented out ones work the same?

    Thanks for the help in advance,
    Wade
    Blackstar, Nov 29, 2006
    #1
    1. Advertising

  2. I have often found the Open Source Ethereal network analysis tool to be
    helpful with this sort of thing:

    http://www.ethereal.com/

    --
    HTH,

    Kevin Spencer
    Microsoft MVP
    Logostician
    http://unclechutney.blogspot.com

    Parabola is a mate of plane.


    "Blackstar" <> wrote in message
    news:...
    >I have a website I am trying to download a file from. This website is
    > strange in the fact that there is no direct link to the URL of the
    > file. The way the site works is as follows:
    >
    > 1) Enter in search criteria
    > 2) Click on Download button to go to download page
    > 3) Click on Download button on Download page
    > 4) Server sends 2 bits of data to a script. The data sent does not
    > include the query data. I'm assuming that the data is stored either in
    > a server side session variable, or server-side on a temporary database.
    > 5) The script processes and initiates a file download. Inside any of
    > the web pages, there are NO direct links directly to a file to
    > download. Once the file is compiled, the webste starts streaming the
    > data to the standard IE file download dialog. I need to capture this
    > file that is downloaded into a stream of some sort, then save it out to
    > a text file. The exported data is a text file.
    >
    > Here is the code I am trying to use:
    >
    > <------ CODE -------->
    > Dim fileURL As New Uri("http://www.somesite.com")
    > Dim myFileWebRequest As WebRequest =
    > WebRequest.Create(fileURL)
    > myFileWebRequest.Method = WebRequestMethods.Http.Post
    > Dim buffer As Byte() =
    > Encoding.UTF8.GetBytes("type=normal&cmd=DOWNLOAD")
    >
    > myFileWebRequest.ContentType =
    > "application/x-www-form-urlencoded"
    > myFileWebRequest.ContentLength = buffer.Length
    > myFileWebRequest.Headers.Add(HttpRequestHeader.Cookie,
    > wb.Document.Cookie)
    > 'myFileWebRequest.Headers.Add(HttpRequestHeader.Referer,
    > "http://www.somesite.com/resource/servlet/search")
    > 'myFileWebRequest.Headers.Add(HttpRequestHeader.Accept,
    > "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
    > application/x-shockwave-flash, application/vnd.ms-excel,
    > application/vnd.ms-powerpoint, application/msword, */*")
    >
    > myFileWebRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us")
    >
    > myFileWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,
    > deflate")
    > 'myFileWebRequest.Headers.Add(HttpRequestHeader.UserAgent,
    > "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
    > 1.1.4322; .NET CLR 2.0.50727)")
    > 'myFileWebRequest.Headers.Add(HttpRequestHeader.Host,
    > "www.somesite.com")
    > 'myFileWebRequest.Headers.Add(HttpRequestHeader.Connection,
    > "Keep-Alive")
    >
    > myFileWebRequest.Headers.Add(HttpRequestHeader.CacheControl,
    > "no-cache")
    >
    > Dim myFileWebResponse As WebResponse =
    > CType(myFileWebRequest.GetResponse, HttpWebResponse)
    > Dim receivestream As Stream =
    > myFileWebResponse.GetResponseStream
    > Dim encode As Encoding =
    > System.Text.Encoding.GetEncoding("utf-8")
    > Dim readStream As New StreamReader(receivestream, encode)
    > Dim read(256) As Char
    > Dim count As Integer = readStream.Read(read, 0, 256)
    > Dim objWriter As New StreamWriter(sFileName)
    >
    > While count > 0
    > Dim str As New String(read, 0, count)
    >
    > objWriter.Write(str)
    > count = readStream.Read(read, 0, 256)
    > End While
    > readStream.Close()
    > objWriter.Close()
    >
    > myFileWebResponse.Close()
    >
    > <--------- END CODE --------->
    >
    > I am not getting ANY errors or exceptions, the program just hangs on
    > the following line:
    > Dim myFileWebResponse As WebResponse =
    > CType(myFileWebRequest.GetResponse, HttpWebResponse)
    >
    > It will sit on this line forever if you let it. The other problem is
    > that the website denies GET requests. It only allows POST requests.
    > I've had a number of other programmers look at it, and they are just as
    > stumped as me. If anyone can help, it would be most appreciated!
    >
    > Also, if you look in the above code, you will notice a large number of
    > lines commented out that are designed to set items in the header. If I
    > uncomment them, then the program hangs and tells me that I am not
    > assigning it correctly and I need to use the name parameter. If the
    > other header items that are uncomment work, then shouldn't the
    > commented out ones work the same?
    >
    > Thanks for the help in advance,
    > Wade
    >
    Kevin Spencer, Nov 30, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Satinderpal Singh

    HttpWebResponse/HttpWebRequest problem...

    Satinderpal Singh, Jun 2, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    3,777
    Satinderpal Singh
    Jun 4, 2004
  2. Sojwal

    HttpWebRequest problem: Please help

    Sojwal, Nov 3, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    450
    Feroze [msft]
    Nov 10, 2004
  3. Darryl
    Replies:
    0
    Views:
    452
    Darryl
    Dec 12, 2003
  4. Satinderpal Singh

    problem with HttpWebRequest/HttpWebResponse...

    Satinderpal Singh, Jun 2, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    4,864
    bruce barker
    Jun 2, 2004
  5. peter
    Replies:
    2
    Views:
    5,998
    Kevin Jones
    Aug 28, 2006
Loading...

Share This Page