Scraping with Nokogiri while using Mechanize

S

Squawk Boxed

The Mechanize documentation says to just start scraping with Nokogiri
once you've navigated to the right page with Mechanize, but this example
doesn't seem to work for me:

agent = Mechanize.new
page = agent.get('http://google.com/')
google_form = page.form('f')
google_form.q = 'ruby mechanize'
page = agent.submit(google_form)

page.xpath('//h3/a[@class="l"]').each do |link|
puts link.content
end

But this works:

page =
Nokogiri::HTML(open('http://www.google.com/search?q=ruby+mechanize'))
page.xpath('//h3/a[@class="l"]').each do |link|
puts link.content
end

Any advice?
 
7

7stud --

Is there a form with the name 'f' on the page www.google.com. According
to the mechanize instructions, you were supposed to do this:

pp page

and look for the form name and form field name you want to fill in.
 
S

Squawk Boxed

The Mechanize part works fine. I navigate to the page I want and when I
pretty print I get the page I want. The problem is I don't know how to
scrape data from the page with Nokogiri once I've navigated to it. For
example, this is how you would do it with pure Nokogiri:

page =
Nokogiri::HTML(open('http://www.google.com/search?q=ruby+mechanize'))
page.xpath('//h3/a[@class="l"]').each do |link|
puts link.content
end


But when I do it with Mechanize, it doesn't output anything.

agent = Mechanize.new
page = agent.get('http://google.com/')
google_form = page.form('f')
google_form.q = 'ruby mechanize'
page = agent.submit(google_form)

page.xpath('//h3/a[@class="l"]').each do |link|
puts link.content
end

It's a trivial example, but for the data I'm scraping I need to use
Mechanize to navigate to the page, so I can't just use Nokogiri. The
thing is the Mechanize documentation says, I should be able to use
regular Nokogiri methods, but they don't seem to work for me.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top