Python Screen Scraper

[James Stroud]

Hello,

Does anyone know of an example, however modest, of a screenscraper
authored in python? I am using Firefox.

Basically, I am answering problems via my browser and being scored for
each problem. I have a tendency to go past my peak for training
efficiency, so I would like to scrape the result page for each problem I
answer, compile statistics, and have a program alert me when I should
stop (based on score and accuracy--assuming training value is related to
changes in these metrics).

I have no idea how to go about writing such a beast and I am hoping that
I could get some pointers or an example that could get me going in the
right direction.

Parsing, etc, is not a problem, but I'm not exactly sure how I might
interface python with Firefox, forwarding scraped pages to my browser
(or forwarding from the browser to the scraper).

Thanks in advance for any help or advice.

James

 

[Michael Bentley]

Hello,

Does anyone know of an example, however modest, of a screenscraper
authored in python? I am using Firefox.

Basically, I am answering problems via my browser and being scored for
each problem. I have a tendency to go past my peak for training
efficiency, so I would like to scrape the result page for each
problem I
answer, compile statistics, and have a program alert me when I should
stop (based on score and accuracy--assuming training value is
related to
changes in these metrics).

I have no idea how to go about writing such a beast and I am hoping
that
I could get some pointers or an example that could get me going in the
right direction.

Parsing, etc, is not a problem, but I'm not exactly sure how I might
interface python with Firefox, forwarding scraped pages to my browser
(or forwarding from the browser to the scraper).

Thanks in advance for any help or advice.

Possibly the easiest thing will be to read from firefox' cache.
Otherwise I think your only real options are to either build a proxy
or sniff the wire...

 

[Jeremy Sanders]

Possibly the easiest thing will be to read from firefox' cache.
Otherwise I think your only real options are to either build a proxy
or sniff the wire...

Maybe another way would be to write a firefox addon/plugin. I believe python
is now supported...

 

[Cameron Laird]

Possibly the easiest thing will be to read from firefox' cache.
Otherwise I think your only real options are to either build a proxy
or sniff the wire...

Oh, there are *lots* of ways to go about this. How crucial is the
role of Firefox in all this? <URL: http://www.greasespot.net/ >
might be the right answer. Are you open to use of languages other
than Python?

 

[kyosohma]

Possibly the easiest thing will be to read from firefox' cache.
Otherwise I think your only real options are to either build a proxy
or sniff the wire...

You should be able to parse the html too. There are quite a few tools
out there for that purpose, "Beautiful Soup" being a good example.

Mike

 

[James Stroud]

Hello,

Does anyone know of an example, however modest, of a screenscraper
authored in python? I am using Firefox.

Basically, I am answering problems via my browser and being scored for
each problem. I have a tendency to go past my peak for training
efficiency, so I would like to scrape the result page for each problem I
answer, compile statistics, and have a program alert me when I should
stop (based on score and accuracy--assuming training value is related to
changes in these metrics).

I have no idea how to go about writing such a beast and I am hoping that
I could get some pointers or an example that could get me going in the
right direction.

Parsing, etc, is not a problem, but I'm not exactly sure how I might
interface python with Firefox, forwarding scraped pages to my browser
(or forwarding from the browser to the scraper).

Thanks in advance for any help or advice.

James

Thanks to everyone for your suggestions.

James

 

[tstrokes]

Possibly the easiest thing will be to read from firefox' cache.
Otherwise I think your only real options are to either build a proxy
or sniff the wire...

You could try twill
http://twill.idyll.org/python-api.html

 

[skotjs]

Hello,

Does anyone know of an example, however modest, of a screenscraper
authored in python? I am using Firefox.

Basically, I am answering problems via my browser and being scored for
each problem. I have a tendency to go past my peak for training
efficiency, so I would like to scrape the result page for each problem I
answer, compile statistics, and have a program alert me when I should
stop (based on score and accuracy--assuming training value is related to
changes in these metrics).

I have no idea how to go about writing such a beast and I am hoping that
I could get some pointers or an example that could get me going in the
right direction.

Parsing, etc, is not a problem, but I'm not exactly sure how I might
interface python with Firefox, forwarding scraped pages to my browser
(or forwarding from the browser to the scraper).

Thanks in advance for any help or advice.

James

Hey James,

This might be of interest to you: http://www.screen-scraper.com/support/docs/scripting_in_python.php

Scott