D
dave.myron
We're about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we're finally
putting a bounty on AOL's head.
Comparing the Firefox session with Mechanize's shows they're nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.
We're offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.
If you'd like a starting point (which might not help much since it
ends up at an error page), here's our code:
=============
require 'rubygems'
require 'mechanize'
agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows IE 6'
page = agent.get( 'http://webmail.aol.com' )
page = agent.get( page.body.scan(/snsRedir\("(https:[^"]
+)/).first.first )
form = page.forms.name('AOLLoginForm').first
form.loginId = '<your username>'
form.password = '<your password>'
page = agent.submit(form, form.buttons.first)
page = agent.get( page.body.scan(/onLoad="checkError[^\)]
+/).first.scan(/'([^']+)'/).last.first )
=============
That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.
Good luck. In addition to your bounty, your work will also net the
community another gem.
Dave
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we're finally
putting a bounty on AOL's head.
Comparing the Firefox session with Mechanize's shows they're nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.
We're offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.
If you'd like a starting point (which might not help much since it
ends up at an error page), here's our code:
=============
require 'rubygems'
require 'mechanize'
agent = WWW::Mechanize.new
agent.user_agent_alias = 'Windows IE 6'
page = agent.get( 'http://webmail.aol.com' )
page = agent.get( page.body.scan(/snsRedir\("(https:[^"]
+)/).first.first )
form = page.forms.name('AOLLoginForm').first
form.loginId = '<your username>'
form.password = '<your password>'
page = agent.submit(form, form.buttons.first)
page = agent.get( page.body.scan(/onLoad="checkError[^\)]
+/).first.scan(/'([^']+)'/).last.first )
=============
That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.
Good luck. In addition to your bounty, your work will also net the
community another gem.
Dave
