Looking for web crawler written in Ruby

[Kev]

------=_Part_2588_26195283.1138691499302
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

Thanks in advance.

Kevin

------=_Part_2588_26195283.1138691499302--

 

[Gene Tani]

Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html
http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf
http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/

http://snippets.textdrive.com/posts/show/74
(look at cached page)

http://rubyforge.org/projects/wee/
look at Mechanize

 

[Justin Collins]

Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

Thanks in advance.

Kevin

If you are _really_ desperate, the first Ruby program I ever wrote was
kind of a web crawler:
http://students.seattleu.edu/collinsj/programs_netcrawler.html

But it's really terrible. It isn't hard to write your own, if you need
to, using Net::HTTP
(http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html) and URI
(http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html).

-Justin

 

[James Edward Gray II]

Hi, I am looking for web crawler(spider) written in Ruby. I googled
but
couldn't find much information. Can anyone give me some links?

The code I translated from Higher-Order Perl at the tail end of the
following blog article may be of use to you.

http://blog.grayproductions.net/articles/2006/01/31/iterators-
chapters-4-and-5

James Edward Gray II

 

[Kev]

------=_Part_4753_29805324.1138795303153
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Many thanks, I appreciate it very much.

------=_Part_4753_29805324.1138795303153--

 

[Bill Guindon]

Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

Some more offerings here:
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/146169?146040=
-146977

now you're practically crawling with spiders

 

[James Britt]

http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html

There's not much spidering going on there, just some page scraping

http://rubyforge.org/projects/wee/
look at Mechanize

My Mechanize example may be helpful, though.

http://neurogami.com/cafe-fetcher/



--
James Britt

http://www.ruby-doc.org - Ruby Help & Documentation
http://www.artima.com/rubycs/ - The Journal By & For Rubyists
http://www.rubystuff.com - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com - Playing with Better Toys
http://www.30secondrule.com - Building Better Tools