Screenscraping, in python, a web page that requires javascript?

[Dan Stromberg - Datallegro]

Is there a method, with python, of screenscraping a web page, if that web
page uses javascript?

I know about BeautifulSoup, but AFAIK at this time, BeautifulSoup is for
HTML that doesn't have embedded javascript.

Thanks!

 

[John J. Lee]

Is there a method, with python, of screenscraping a web page, if that web
page uses javascript?

Not pure CPython, no.

I know about BeautifulSoup, but AFAIK at this time, BeautifulSoup is for
HTML that doesn't have embedded javascript.

It's not that BeautifulSoup is unhappy with JS, it's just that there's
no support for executing the JS.

There are some Java libraries that know how to execute JS embedded in
web pages, which could be used from Jython:

http://www.thefrontside.net/crosscheck

http://htmlunit.sourceforge.net/

http://httpunit.sourceforge.net/


You can also automate a browser, but that still seems to be painful in
one way or another.


John