Using Mechanize To Submit Forms

R

Robert Molloy

Hi guys,

I'm trying to use Mechanize with Ruby to scrape data from this site
(http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do). The first part of my
task is trying to automate an initial search form, I can do this fine
using the following code.


require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do')

search_form = page.form('HPLCDS0101Form')
search_form.callJorEFlg = 1
search_form['method'] = 'search'
search_form.exeKind = 'HPLCDS0101E'
search_form.searchListCount = 2500
search_form.checkbox_with:)value => '001').check
search_form.checkbox_with:)value => '002').check
search_form.checkbox_with:)value => '004').check

page = agent.submit(search_form)
stockform = page.form('HPLCDS0301Form')

stock = stockform.button_with:)value => 'Display of stock price')

pp stock


When I submit the form, the data I get returned contains over 2300 links
of the type below. The onclick="chart('1378')" with the number is the
stock ticker symbol that determines which stock is going to be opened
when you click that particular link.

<input type="button" property="chart_button" value="Display of stock
price" onclick="chart('1378')" class="negativeButton" />

Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.

Can anyone help nudge me along here? Really appreciate it, thanks!
 
G

gf

Mechanize can return a Nokogiri::HTML document when you do:

doc =3D page.parser

Using Nokogiri you can do something like:

doc.css('input[@class=3D"negativeButton"]')

to retrieve an array of the input tags with a class of
"negativeButton". You can fine-tune the returned values as needed
using select() or reject() or by switching from .css() to .xpath().

You'll need to look at the contents of the chart() Javascript function
which is being called by the onclick() handler to figure out what the
add the parameter to, and that will give you the full URL to retrieve.
 
R

Ryan Davis

<input type=3D"button" property=3D"chart_button" value=3D"Display of = stock
price" onclick=3D"chart('1378')" class=3D"negativeButton" />
=20
Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.

You're prolly looking for something like:

search_form.buttons_with:)onclick =3D> /^chart/).each do |button|
button.click
end

this is untested, but a good portion was glommed from mechanize's tests. =
Do this:

gem unpack mechanize
cd mechanize*
grep -r buttons_with .

and such. I find lots of stuff that way.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top