web scrapping - POST and auto-login

Discussion in 'Python' started by james, Sep 20, 2005.

  1. james

    james Guest

    please excuse my ignorance of Python. i know enough to be dangerous
    (was able however to make Freevo support my FM tuner with a little hacking).

    Ultimate goal:
    read N bytes from PATTERN on page A into VARIABLE.
    POST VARIABLE into TABLE on site B.
    email VARIABLE to myself.

    what i have to work with:
    ---------------------------
    #!/usr/local/bin/python

    from urllib import urlopen
    from re import search

    URL = "http://wifirouter/WL_WPATable.asp"
    pattern = "wl_key1 value='*([0-9,a-f,]*)"
    doc = urlopen(URL).read()
    result = search(pattern, doc)
    print result.group(1)
    ---------------------------

    I'm trying to update the WEP key on a wireless router via script and
    email the results to myself. this will be run once a week. Site A
    already generates random keys, so i figure that's a good place to get
    VARIABLE from. The example above gives me the current key of the router.
    What i'm having trouble with is the key on site A (one i wrote myself
    http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi ) provides 128bit
    keys by default, but i only want to use 64bit (10 characters). so..
    rather than make it select 64bit and all i figure it would be easier to
    use the above example for getting the /current/ key, modified it to read
    the entire 26 character string from site A, but [1] how do i crop it to
    10 characters.
    [2] how can i then auto-login to site B (the router's web config via
    pop-up dialog box. *not* a form) and then [3] post VARIABLE and save the
    changes.

    I know this sounds like a lot, but if anyone can provide me with any
    example code i can maybe modify to fit, or point me to some good
    reference material. all the pages i found on logging into a site
    involve ssl and cookies, which, in this case at least, isn't involved here.
    If there is any other info i can provide (ie. sections of relevent
    HTML) then let me know. any assistance you have to offer is greatly
    appreciated.
     
    james, Sep 20, 2005
    #1
    1. Advertisements

  2. james

    james Guest

    [1] still at a loss on this one, but i can get to it later, unless you've got
    any ideas.
    ok.. this one was stupid easy. should have remembered that.
    [2] http://user:[email protected]/WL_WPATable.asp
    argh!
    [3]
    now i'm trying to get it to find the BUTTON "Save Settings". from what i see in
    the code it's labled "save_button", but i only get:
    ------------------------
    ....
    ....
    <IgnoreControl(save_button"=<None>)>
    <IgnoreControl(cancel"=<None>)>>
    Traceback (most recent call last):
    File "./wifi.py", line 19, in ?
    response = urlopen(form.click("save_button"))
    File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2628, in click
    self._request_class)
    File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2788, in _click
    control = self._find_control(name, type, "clickable", id, None, nr)
    File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2783, in
    _find_control
    raise ControlNotFoundError("no control matching "+description)
    ClientForm.ControlNotFoundError: no control matching name 'save_button', kind
    'clickable'
    ---------------------------

    relevent HTML:
    ---------------------------
    ....
    ....
    <TD
    height=25>&nbsp;<script>Capture(wlansec.key1)</script>&nbsp;:&nbsp;</TD>
    <TD height=25>&nbsp;<INPUT size=36 name=wl_key1 value='5465a6576b'></TD>
    ....
    ....
    <script>document.write("<input type=button name=save_button" + " value=\"" +
    sbutton.save + "\" onClick=to_submit(this.form)>");</script>&nbsp;
    ....
    ....
     
    james, Sep 20, 2005
    #2
    1. Advertisements

  3. james

    Max Erickson Guest

    strings are slicable:
    As far as your other problem, it might make sense(I don't know...) to
    just generate the post request for saving the new password yourself
    and never bother with parsing/filling the form.

    max
     
    Max Erickson, Sep 20, 2005
    #3
  4. Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the
    more low-level ClientForm by the same author. This will be _much_
    easier than doing it by hand.

    -Jonathan
     
    Jonathan Ellis, Sep 20, 2005
    #4
  5. james

    james Guest

    what's the fun in that though? :)
    actually, i was just looking at that site, which is where i came across
    ClientForm. i think it's getting confused.. picks up "save_button", but lists
    it as "<IgnoreControl(save_button"=<None>)>"

    the stray " is from:
    <script>document.write("<input type=button name=save_button" + " value=\".....

    if i can just get it to see that correctly, then i think i can make it work from
    there. any ideas on that?
     
    james, Sep 20, 2005
    #5
  6. james

    james Guest

    The only reason i've gotten this far is a basic understanding of syntax and
    programming in general :)
    care to enlighten me a little on how to do that?
    i'm not quite sure how to generate it from scratch.
    thanks for the input/advice!
     
    james, Sep 20, 2005
    #6
  7. james

    james Guest

    wep64 = wep128[0:10]

    sorry for the stupid question. thanks.
     
    james, Sep 20, 2005
    #7
  8. james

    Max Erickson Guest

    I did. If you have some text in VARIABLE, and do something like:
    substring=VARIABLE[:10], substring will be the first 10 characters of
    VARIABLE.
     
    Max Erickson, Sep 20, 2005
    #8
  9. james

    james Guest

    What i have so far:
    --------------------
    #!/usr/local/bin/python

    from urllib import urlopen
    from ClientForm import ParseResponse
    from re import search

    URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
    pattern = "<!-- marker -->*([0-9,a-f,]*)"
    doc = urlopen(URL).read()
    result = search(pattern, doc)
    wep64 = result.group(1)[:10]
    #print wep64

    forms = ParseResponse(urlopen("http://un:[email protected]/WL_WPATable.asp"))
    form = forms[0]
    form["wl_key1"] = wep64
    #print form
    response = urlopen(form.click("save_button"))

    URL = "http://un:[email protected]/WL_WPATable.asp"
    pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
    doc2 = urlopen(URL).read()
    result2 = search(pattern2, doc2)
    #print result2.group(1)
    ---------------------------

    the only problems now are getting it to see the "save_button" correctly or
    generate the POST from scratch as Max suggested.. that will take some more
    hunting to figure that out... any better phrases to search for maybe?
    and email it out. can probably "borrow" some things from Freevo on calling
    another program, just passing the variable to it as a PIPE is going to be the
    hard part. thinking of using sendmail (not a problem), but how to get the data?
    i'll keep looking. thanks again both of you.
     
    james, Sep 20, 2005
    #9
  10. james

    james Guest

    What i have so far:
    --------------------
    #!/usr/local/bin/python

    from urllib import urlopen
    from ClientForm import ParseResponse
    from re import search

    URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
    pattern = "<!-- marker -->*([0-9,a-f,]*)"
    doc = urlopen(URL).read()
    result = search(pattern, doc)
    wep64 = result.group(1)[:10]
    #print wep64

    forms = ParseResponse(urlopen("http://un:[email protected]/WL_WPATable.asp"))
    form = forms[0]
    form["wl_key1"] = wep64
    #print form
    response = urlopen(form.click("save_button"))

    URL = "http://un:[email protected]/WL_WPATable.asp"
    pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
    doc2 = urlopen(URL).read()
    result2 = search(pattern2, doc2)
    #print result2.group(1)
    ---------------------------

    the only problems now are getting it to see the "save_button" correctly or
    generate the POST from scratch as Max suggested.. that will take some more
    hunting to figure that out... any better phrases to search for maybe?
    and email it out. can probably "borrow" some things from Freevo on calling
    another program, just passing the variable to it as a PIPE is going to be the
    hard part. thinking of using sendmail (not a problem), but how to get the data?
    i'll keep looking. thanks again both of you.
     
    james, Sep 20, 2005
    #10
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.