General Web Scraping Question

Discussion in 'ASP .Net' started by Selden McCabe, Feb 22, 2004.

  1. I've been working on a web scraping program, and have the basics down.

    But I don't understand the parameters.
    Normally, you go to a URL (say a reverse yellow pages directory), and enter
    some parameters (like area code, phone number, etc.) and POST this back to
    the web. Then you parse the response, looking for the data you need.

    Ofen I see examples where the data you post contains something like
    "AreaCode=503&Number=5551212&x=1&y=2"

    Where do the "x=1 and y=2" come from? I have some sites where my post
    doesn't work. In one case, you are supposed to enter a contractor's license
    number, and then click a button, and the result contains information about
    the license. After I post what I think should work, the result coming back
    is the same web page, with the contractor's number filled in.

    Do the X and Y parameters involve invoking a button? How do you determine
    what to use for the parameters?

    Thanks in advance for any advice or pointers!
    ---Selden McCabe
     
    Selden McCabe, Feb 22, 2004
    #1
    1. Advertising

  2. I suspect X and Y are passed by the browser when the user clicks on an image
    map. Have you tried passing &x=1&y=1 in your post?

    --
    Thanks,

    Eric Lawrence
    Program Manager
    Assistance and Worldwide Services

    This posting is provided "AS IS" with no warranties, and confers no rights.


    "Selden McCabe" <> wrote in message
    news:#lFAcCO#...
    > I've been working on a web scraping program, and have the basics down.
    >
    > But I don't understand the parameters.
    > Normally, you go to a URL (say a reverse yellow pages directory), and

    enter
    > some parameters (like area code, phone number, etc.) and POST this back to
    > the web. Then you parse the response, looking for the data you need.
    >
    > Ofen I see examples where the data you post contains something like
    > "AreaCode=503&Number=5551212&x=1&y=2"
    >
    > Where do the "x=1 and y=2" come from? I have some sites where my post
    > doesn't work. In one case, you are supposed to enter a contractor's

    license
    > number, and then click a button, and the result contains information about
    > the license. After I post what I think should work, the result coming

    back
    > is the same web page, with the contractor's number filled in.
    >
    > Do the X and Y parameters involve invoking a button? How do you determine
    > what to use for the parameters?
    >
    > Thanks in advance for any advice or pointers!
    > ---Selden McCabe
    >
    >
     
    Eric Lawrence [MSFT], Feb 22, 2004
    #2
    1. Advertising

  3. Selden McCabe

    Joerg Jooss Guest

    Selden McCabe wrote:
    > I've been working on a web scraping program, and have the basics down.
    >
    > But I don't understand the parameters.
    > Normally, you go to a URL (say a reverse yellow pages directory), and
    > enter some parameters (like area code, phone number, etc.) and POST
    > this back to the web. Then you parse the response, looking for the
    > data you need.
    >
    > Ofen I see examples where the data you post contains something like
    > "AreaCode=503&Number=5551212&x=1&y=2"
    >
    > Where do the "x=1 and y=2" come from? I have some sites where my post
    > doesn't work. In one case, you are supposed to enter a contractor's
    > license number, and then click a button, and the result contains
    > information about the license. After I post what I think should
    > work, the result coming back is the same web page, with the
    > contractor's number filled in.
    >
    > Do the X and Y parameters involve invoking a button? How do you
    > determine what to use for the parameters?


    These could be hidden fields used by web application to store session state
    on the client. Actually, it's not easy to implement web scraping for
    "foreign" web applications where you don't have access to the code or at
    least some inside knowledge.

    Cheers,

    --
    Joerg Jooss
     
    Joerg Jooss, Feb 22, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Selden McCabe

    Web Scraping Question

    Selden McCabe, Feb 21, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    353
    Chad Z. Hower aka Kudzu
    Feb 21, 2004
  2. Brent
    Replies:
    8
    Views:
    8,386
    Developer2010
    Oct 19, 2010
  3. Lucas Tam

    Web Screen Scraping Tools

    Lucas Tam, Oct 6, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    657
    Paul D. Fox
    Oct 6, 2005
  4. David Jones

    Web Scraping/Site Scraping

    David Jones, Jul 11, 2004, in forum: Python
    Replies:
    4
    Views:
    506
    Andrew Bennetts
    Jul 13, 2004
  5. KMA

    General Q about screen scraping

    KMA, Oct 20, 2005, in forum: ASP .Net Web Services
    Replies:
    3
    Views:
    143
Loading...

Share This Page