How to imitate a browser?

[Marek Osvald]

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?

 

[Jesús Gabriel y Galán]

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?

Use Mechanize:

http://mechanize.rubyforge.org/mechanize/

It manages cookies and allows you to fill forms, click links and so on.

Jesus.

 

[Adriano Ferreira]

[Note: parts of this message were removed to make it a legal post.]

Hi Marek,

Have you tried the combo Nokogiri (http://nokogiri.org/) / Mechanize
(http://mechanize.rubyforge.org/mechanize/) to do the job?

I recommend you to take a look at:
* http://railscasts.com/episodes/190-screen-scraping-with-nokogiri
* http://railscasts.com/episodes/191-mechanize
Adriano



________________________________
From: Marek Osvald <[email protected]>
To: ruby-talk ML <[email protected]>
Sent: Sun, October 3, 2010 11:28:38 AM
Subject: How to imitate a browser?

Hey guys,

so I'm trying to write a simple script for extraction of my profile data
from one site. In order to get the data I have to log in and then
download a page.

At first I tried to use Ruby's Net::HTTP library, but the problem is
that the site runs PHP and logs users in via storing cookies (therefore
the low-level network approach doesn't work).

Any ideas?

 

[Roger Pack]

so I'm trying to write a simple script for extraction of my profile data

from one site.

http://ruby-toolbox.com/categories/http_clients.html
http://ruby-toolbox.com/categories/browser_testing.html (watir,
especially)