Screen scraping, then caching.... not working on web parts page?

Discussion in 'ASP .Net' started by, Jun 25, 2007.

  1. Guest

    We have a web parts intranet at our office here I have been working

    I have been trying to create a way of getting our OCE TDS 600 plotter
    queue to be a web part on the page, seeing as their is no way of
    interfacing apart from their 'print exec workgroup' web based program.

    So I decided to screen scrape, I've done a scrape of the intranet page
    of print exec workgroup (http://tds600 on our network) seemed to go
    fine, then i used regex, and some replace's to delete the stuff I
    didn't want and change some formatting to suit my page then wrote the
    html code to a label, seemed to work fine.

    I then decided maybe I should do some caching, as this plotter doesnt
    want to be hit up when ever user in the network (with autorefresh
    every 5 mins) as it is receiving drawings and stuff.

    So I tried caching it just for a minute or two, just so that way only
    one person hits it every 2 minutes - the rest get it from cache.

    But it doesn't seem to be working, I made a test page, seemed to cache
    fine, but then when I made it an ascx control and put it on my web
    parts page, it never caches. And perhaps as a result, actually
    retreiving the scrape is a hit and miss afair, and even worse, people
    can't access print exec (which is used to send drawings to plotter) as
    it says theres too many users connected (theres no license limit - but
    it's obviously a hard limit set so that bandwidth for plots doesnt
    suffer - so the web connections must be making lots and overloading

    I'm making two parts, 1 to get the status on rolls (empty icons etc)
    and one for queue, they are essentiall they same.

    Code (Text):

     Sub Page_Load(sender as Object, e as EventArgs)
            Dim RollStatus
            Const strURL As String = "http://tds600/servlet/PEBServlet?

            If Cache.Get("RollState") IsNot Nothing Then
                RollStatus = Cache.Get("RollState")
                    Response.Write("retrieved from cache")
                End If

               response.write ("Regenerating Data")
                Dim PrintExecPage As String
                Dim req As HttpWebRequest = WebRequest.Create(strURL)
                req.Timeout = 1000

                Dim resp As HttpWebResponse = req.GetResponse()

                Dim sr As New StreamReader(resp.GetResponseStream())
                PrintExecPage = sr.ReadToEnd()

                Dim RegEx As New Regex("<table width='100%' border='0'
    cellpadding='2' cellspacing='0'><tr>(.*?)Feeder</td>(.*?)</table>",

                Dim objmatch As Match

                For Each objmatch In RegEx.Matches(PrintExecPage)

                   'filter out the shit, store it in RollStatus string


                If Len(RollStatus) > 20 Then
                    Cache.Insert("RollState", RollStatus, Nothing,
    DateTime.Now.AddMinutes(3), TimeSpan.Zero)
                   response.write ("Cache Written")

                End If


            End If

            If RollStatus Is Nothing Then
                RollStatus = "Did not load."
            End If

            lblHTMLOutput.Text = RollStatus

        End Sub
    whenever the stream actually loads, it tells me its regenerating data,
    and that its written to cache, then I hit refresh, and it says the
    same thing - no recall from cache.

    Any ideas? what have I done wrong? Is it possible our web server is
    deleting cache, is there any way to check?
    , Jun 25, 2007
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lucas Tam

    Web Screen Scraping Tools

    Lucas Tam, Oct 6, 2005, in forum: ASP .Net
    Paul D. Fox
    Oct 6, 2005
  2. David Jones

    Web Scraping/Site Scraping

    David Jones, Jul 11, 2004, in forum: Python
    Andrew Bennetts
    Jul 13, 2004
  3. Replies:
    Apr 12, 2007
  4. Replies:
  5. Replies:

Share This Page