Automatic web form transactions

[Jay Davis]

We've been using the excellent ClientForms module with
urllib2 to read forms, fill them in, and then submit
with 'response = urlopen(form.click())'. The process
usually works very well.

But some forms, which work straightforward in a
browser, don't work with form.click(). For instance,
in one simple form the response is:

"Sorry, this form cannot be posted to from external source."

I'm wondering if the reason it generates that response is
because it is looking for a standard browser. In that
case, I should be able to send some browser string that it likes.
I don't think I can use the webbrowser module because
while that lets you launch a browser, it doesn't have an
interface to fill in forms, click the submit button, and so on.

One obvious long term solution is for me to learn much
more about the details of html form processing, especially
low level details in the htmllib and ClientForm modules.
But a nice quick solution would be helpful too.

If its any use, the offending form is:

<form method=post action=/additem.php name=TaskForm>
<table border=0 cellpadding=4 cellspace=4>
<tr>
<td bgcolor=000090 width=95 align=left>
<font face=arial size=2 color=ffffff>
New Loc
</font>
</td>
<td bgcolor=000090 align=left>
<font size=2 face=arial color=ffffff>
<input type=text name=newLoc value="http://" size=25 maxlength=255>
</td>
<td bgcolor=000090 align=center colspan=2>
<font size=1 face=arial>
<input type=submit value="Submit Form">
<input type=reset value="Clear Form">
</td>
</tr>
</table>
<form>

Thanks,

J

 

[DH]

We've been using the excellent ClientForms module with
urllib2 to read forms, fill them in, and then submit
with 'response = urlopen(form.click())'. The process
usually works very well.

But some forms, which work straightforward in a
browser, don't work with form.click(). For instance,
in one simple form the response is:

"Sorry, this form cannot be posted to from external source."

The web form script (additem.php) might be checking the referrer.
Set the http request referrer header to a URL the the URL of the webform.

See if something like this works:
request = form.click()
request.add_header("Referrer", "url of the form")
response = urlopen(request)

If that doesn't work, maybe it is checking a cookie or something.
You could replace urlopen with ClientCookie.urlopen maybe.

 

[Jay Davis]

The web form script (additem.php) might be checking the referrer.
Set the http request referrer header to a URL the the URL of the webform.

See if something like this works:
request = form.click()
request.add_header("Referrer", "url of the form")
response = urlopen(request)

Yep that was it. (But replace('Referrer','Referer').)

Thanks!