Need to parse a HTML table into an Array/hash

[Ernst Tanaka]

I am new to ruby; having a lot of fun, but currently I am stuck with the
following.

I want to read this page

http://finance.yahoo.com/q/op?s=AAPL&k=95.00&m=2007-12

Into an array or hash.

I found the gem yahoo-finance; but that does not what I need.
I need to find the Symbol for (in this example)
AAPL call option strike 95 Dec-07.



I am able to read the page:
conn = open("http://finance.yahoo.com/q/op?s=AAPL&m=2008-04")
lines = conn.read

The main question is how to I parse the table which is in the middle of
the HTML.

Any help is appreciated!!

 

[Ball, Donald A Jr (Library)]

I am able to read the page:

conn =3D open("http://finance.yahoo.com/q/op?s=3DAAPL&m=3D2008-04")
lines =3D conn.read
=20
The main question is how to I parse the table which is in the=20
middle of the HTML.

You probably want to look at mechanize and hpricot, or maybe scrubyt.

- donald

 

[Dejan Dimic]

You probably want to look at mechanize and hpricot, or maybe scrubyt.

- donald

I use hpricot for the similar task.

 

[Ruby Ronin]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

I'm kind of new too, so this might be a little sloppy by more experienced
standards, but I think this gets you partially there.

require "open-uri"
conn = open("http://finance.yahoo.com/q/op?s=AAPL&m=2008-04").read
table_data = conn.scan /<td.+?td>/
table_data_refined = []
table_data.each { |data|
data.gsub!(/<.+?>/, "")
table_data_refined << data
}

You probably need to tweak the regexes to isolate the specific table you
want (like plugging in "class=xxx").

I hope that helps.

 

[Peter Szinek]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

I am new to ruby; having a lot of fun, but currently I am stuck with
the
following.

I want to read this page

http://finance.yahoo.com/q/op?s=AAPL&k=95.00&m=2007-12

scRUBYt! is especially well suited for this type of task.

Watch this:

==================================================
require 'rubygems'
require 'scrubyt'

Scrubyt.logger = Scrubyt::Logger.new
yahoo_data = Scrubyt::Extractor.define do

fetch ' http://finance.yahoo.com/q/op?s=AAPL&k=95.00&m=2007-12'

record do
expires 'Dec 07'
symbol 'QAALS.X'
last '86.60'
chg 'Down 3.00'
bid '87.10'
ask '87.50'
vol '4'
open_int '146'
end.select_indices(1..6)
end

p yahoo_data.to_hash
yahoo_data.export(__FILE__)
==================================================
If you want to get just the first 3 results, change select_indices(1..3)

output:
==================================================
[{:chg=>"Dec 07QAALS.X86.60 3.0087.1087.504146",
:bid=>"87.10", :ask=>"87.50", :vol=>"4", :last=>"86.60",
:expires=>"Dec 07", pen_int=>"146", :symbol=>" QAALS.X"},
{:chg=>"Jan 08QAAAS.X88.10 0.9587.4587.901714,832",
:bid=>"87.45", :ask=>"87.90", :vol=>"17", :last=>"88.10",
:expires=>"Jan 08", pen_int=>"14,832", :symbol=>" QAAAS.X"},
{:chg=>"Apr 08QAADS.X88.35 2.1089.0089.4511973",
:bid=>"89.00", :ask=>"89.45", :vol=>"11", :last=>"88.35",
:expires=>"Apr 08", pen_int=>"973", :symbol=>" QAADS.X"},
{:chg=>"ExpiresSymbolLastChgBidAskVolOpen Int", :bid=>"Bid",
:ask=>"Ask", :vol=>"Vol", pen_int=>"Open Int"},
{:chg=>"Dec 07QAAXS.X0.01 0.00N/A0.034128", :bid=>"N/A",
:ask=>"0.03", :vol=>"4", :last=>"0.01", :expires=>"Dec 07",
pen_int=>"128", :symbol=>"QAAXS.X"},
{:chg=>"Jan 08QAAMS.X0.10 0.040.070.112,38619,345",
:bid=>"0.07", :ask=>"0.11", :vol=>"2,386", :last=>"0.10",
:expires=>"Jan 08", pen_int=>"19,345", :symbol=>" QAAMS.X"}]
==================================================

The very last line also creates a second file, which looks like this:

==================================================
require 'rubygems'
require 'scrubyt'

yahoo_data = Scrubyt::Extractor.define do
fetch(" http://finance.yahoo.com/q/op?s=AAPL&k=95.00&m=2007-12")

record("/html/body/div/div/table/tr/td/table/tr/td/table/tr",
{ :generalize => true }) do
expires("/td[1]/b[1]/a[1]")
symbol("/td[2]/a[1]")
last("/td[3]/b[1]")
chg()
bid("/td[5]")
ask("/td[6]")
vol("/td[7]")
open_int("/td[8]")
end.select_indices((1..6))
end

yahoo_data.to_xml.write($stdout, 1)
==================================================

which obviously works whatever the content of the page is.

There is much more in sRUBYt!, this was just a very basic example, so
if you need some aditional tweaking, just drop me a mail.

HTH,
Peter
___
http://www.rubyrailways.com
http://scrubyt.org

 

[Ernst Tanaka]

Peter Szinek wrote:




Peter:

Thanks for your help.

Your method works nice and easy.

Although installing of scrubyt took a little effort.
I encountered some errors // but as adviced on a this website
http://agora.scrubyt.org/forums/3/topics/60?page=2
I made a comment of require ‘parse_tree_reloaded’

Thanks to everyone; the help I get on this forum makes programming Ruby
a very pleasant experience.

Ernst

 

[Vitor Peres]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

I should say, even if a bit late, that Yahoo! Finance provides data in CSV
format -- you can specify a start and end date, and it gives you the
closings for each day in the range. It's easier than screen scraping.

Historical data can be obtained with the following URL:

http://ichart.finance.yahoo.com/table.csv?&s=[quote name]&a=[start
month]&b=[start_day]&c=[start_year]&d=[end_month]&e=[end_day]&f=[end_year]&g=d&ignore=.csv

Substitute each bracketed info with the corresponding number. Leave the
brackts out, of course.

You can get data for the current quote in:

http://finance.yahoo.com/d/quotes.csv?s=[quote name]&f=sl1d1t1c1ohgv&e=.csv

Just put in the name of the thingie and you'll be fine.

-Vitor

 

[Ernst Tanaka]

Thanks Vitor; But for that solution you need to know the Symbol.
My problem is I have the symbol of the underlying and the Strike of the
option.
I don't have the symbol for the option.
That why I need to parse the option page to get the symbol and then to
move to that cvs file.

Thank you for your response.

Ernst

 

[Vitor Peres]

Note: parts of this message were removed by the gateway to make it a legal Usenet post.

Thanks Vitor; But for that solution you need to know the Symbol.
My problem is I have the symbol of the underlying and the Strike of the
option.
I don't have the symbol for the option.
That why I need to parse the option page to get the symbol and then to
move to that cvs file.

Thank you for your response.

Ernst

Oh, I'm sorry. I seem to have lacked in attention when reading your original
message.