web scrapping - POST and auto-login

[james]

please excuse my ignorance of Python. i know enough to be dangerous
(was able however to make Freevo support my FM tuner with a little hacking).

Ultimate goal:
read N bytes from PATTERN on page A into VARIABLE.
POST VARIABLE into TABLE on site B.
email VARIABLE to myself.

what i have to work with:
---------------------------
#!/usr/local/bin/python

from urllib import urlopen
from re import search

URL = "http://wifirouter/WL_WPATable.asp"
pattern = "wl_key1 value='*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
print result.group(1)
---------------------------

I'm trying to update the WEP key on a wireless router via script and
email the results to myself. this will be run once a week. Site A
already generates random keys, so i figure that's a good place to get
VARIABLE from. The example above gives me the current key of the router.
What i'm having trouble with is the key on site A (one i wrote myself
http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi ) provides 128bit
keys by default, but i only want to use 64bit (10 characters). so..
rather than make it select 64bit and all i figure it would be easier to
use the above example for getting the /current/ key, modified it to read
the entire 26 character string from site A, but [1] how do i crop it to
10 characters.
[2] how can i then auto-login to site B (the router's web config via
pop-up dialog box. *not* a form) and then [3] post VARIABLE and save the
changes.

I know this sounds like a lot, but if anyone can provide me with any
example code i can maybe modify to fit, or point me to some good
reference material. all the pages i found on logging into a site
involve ssl and cookies, which, in this case at least, isn't involved here.
If there is any other info i can provide (ie. sections of relevent
HTML) then let me know. any assistance you have to offer is greatly
appreciated.

 

[james]

the entire 26 character string from site A, but [1] how do i crop it to
10 characters.

[1] still at a loss on this one, but i can get to it later, unless you've got
any ideas.

[2] how can i then auto-login to site B (the router's web config via
pop-up dialog box. *not* a form) and then [3] post VARIABLE and save the
changes.

ok.. this one was stupid easy. should have remembered that.
[2] http://user:password@wifirouter/WL_WPATable.asp

If there is any other info i can provide (ie. sections of relevent
HTML) then let me know. any assistance you have to offer is greatly
appreciated.

argh!
[3]
now i'm trying to get it to find the BUTTON "Save Settings". from what i see in
the code it's labled "save_button", but i only get:
------------------------
....
....
<IgnoreControl(save_button"=<None>)>
<IgnoreControl(cancel"=<None>)>>
Traceback (most recent call last):
File "./wifi.py", line 19, in ?
response = urlopen(form.click("save_button"))
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2628, in click
self._request_class)
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2788, in _click
control = self._find_control(name, type, "clickable", id, None, nr)
File "/usr/local/lib/python2.4/site-packages/ClientForm.py", line 2783, in
_find_control
raise ControlNotFoundError("no control matching "+description)
ClientForm.ControlNotFoundError: no control matching name 'save_button', kind
'clickable'
---------------------------

relevent HTML:
---------------------------
....
....
<TD
height=25>&nbsp;<script>Capture(wlansec.key1)</script>&nbsp;:&nbsp;</TD>
<TD height=25>&nbsp;<INPUT size=36 name=wl_key1 value='5465a6576b'></TD>
....
....
<script>document.write("<input type=button name=save_button" + " value=\"" +
sbutton.save + "\" onClick=to_submit(this.form)>");</script>&nbsp;
....
....

 

[Max Erickson]

the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.

[1] still at a loss on this one, but i can get to it later,
unless you've got any ideas.

strings are slicable:

s='a string of characters'
s[:10] 'a string o'
s[-10:] 'characters'

As far as your other problem, it might make sense(I don't know...) to
just generate the post request for saving the new password yourself
and never bother with parsing/filling the form.

max

 

[Jonathan Ellis]

I'm trying to update the WEP key on a wireless router via script and
email the results to myself. this will be run once a week.

Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the
more low-level ClientForm by the same author. This will be _much_
easier than doing it by hand.

-Jonathan

 

[james]

Look up Mechanize (http://wwwsearch.sourceforge.net/mechanize/) or the
more low-level ClientForm by the same author. This will be _much_
easier than doing it by hand.

-Jonathan

what's the fun in that though?
actually, i was just looking at that site, which is where i came across
ClientForm. i think it's getting confused.. picks up "save_button", but lists
it as "<IgnoreControl(save_button"=<None>)>"

the stray " is from:
<script>document.write("<input type=button name=save_button" + " value=\".....

if i can just get it to see that correctly, then i think i can make it work from
there. any ideas on that?

 

[james]

the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.

strings are slicable:

The only reason i've gotten this far is a basic understanding of syntax and
programming in general
care to enlighten me a little on how to do that?

As far as your other problem, it might make sense(I don't know...) to
just generate the post request for saving the new password yourself
and never bother with parsing/filling the form.

i'm not quite sure how to generate it from scratch.
thanks for the input/advice!

 

[james]

strings are slicable:

wep64 = wep128[0:10]

sorry for the stupid question. thanks.

 

[Max Erickson]

the entire 26 character string from site A, but [1] how do i
crop it to 10 characters.

strings are slicable:

The only reason i've gotten this far is a basic understanding of
syntax and programming in general
care to enlighten me a little on how to do that?

I did. If you have some text in VARIABLE, and do something like:
substring=VARIABLE[:10], substring will be the first 10 characters of
VARIABLE.

 

[james]

What i have so far:
--------------------
#!/usr/local/bin/python

from urllib import urlopen
from ClientForm import ParseResponse
from re import search

URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
pattern = "<!-- marker -->*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
wep64 = result.group(1)[:10]
#print wep64

forms = ParseResponse(urlopen("http://un:pw@wifirouter/WL_WPATable.asp"))
form = forms[0]
form["wl_key1"] = wep64
#print form
response = urlopen(form.click("save_button"))

URL = "http://un:pw@wifirouter/WL_WPATable.asp"
pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
doc2 = urlopen(URL).read()
result2 = search(pattern2, doc2)
#print result2.group(1)
---------------------------

the only problems now are getting it to see the "save_button" correctly or
generate the POST from scratch as Max suggested.. that will take some more
hunting to figure that out... any better phrases to search for maybe?
and email it out. can probably "borrow" some things from Freevo on calling
another program, just passing the variable to it as a PIPE is going to be the
hard part. thinking of using sendmail (not a problem), but how to get the data?
i'll keep looking. thanks again both of you.

 

[james]

What i have so far:
--------------------
#!/usr/local/bin/python

from urllib import urlopen
from ClientForm import ParseResponse
from re import search

URL = "http://www.hal-pc.org/networking/cgi-bin/wepgen.cgi"
pattern = "<!-- marker -->*([0-9,a-f,]*)"
doc = urlopen(URL).read()
result = search(pattern, doc)
wep64 = result.group(1)[:10]
#print wep64

forms = ParseResponse(urlopen("http://un:pw@wifirouter/WL_WPATable.asp"))
form = forms[0]
form["wl_key1"] = wep64
#print form
response = urlopen(form.click("save_button"))

URL = "http://un:pw@wifirouter/WL_WPATable.asp"
pattern2 = "wl_key1 value='*([0-9,a-f,]*)"
doc2 = urlopen(URL).read()
result2 = search(pattern2, doc2)
#print result2.group(1)
---------------------------

the only problems now are getting it to see the "save_button" correctly or
generate the POST from scratch as Max suggested.. that will take some more
hunting to figure that out... any better phrases to search for maybe?
and email it out. can probably "borrow" some things from Freevo on calling
another program, just passing the variable to it as a PIPE is going to be the
hard part. thinking of using sendmail (not a problem), but how to get the data?
i'll keep looking. thanks again both of you.