Looking for web crawler written in Ruby

Discussion in 'Ruby' started by Kev, Jan 31, 2006.

  1. Kev

    Kev Guest

    ------=_Part_2588_26195283.1138691499302
    Content-Type: text/plain; charset=ISO-8859-1
    Content-Transfer-Encoding: quoted-printable
    Content-Disposition: inline

    Hi, I am looking for web crawler(spider) written in Ruby. I googled but
    couldn't find much information. Can anyone give me some links?

    Thanks in advance.

    Kevin

    ------=_Part_2588_26195283.1138691499302--
    Kev, Jan 31, 2006
    #1
    1. Advertising

  2. Kev

    Gene Tani Guest

    Gene Tani, Jan 31, 2006
    #2
    1. Advertising

  3. Kev wrote:
    > Hi, I am looking for web crawler(spider) written in Ruby. I googled but
    > couldn't find much information. Can anyone give me some links?
    >
    > Thanks in advance.
    >
    > Kevin
    >


    If you are _really_ desperate, the first Ruby program I ever wrote was
    kind of a web crawler:
    http://students.seattleu.edu/collinsj/programs_netcrawler.html

    But it's really terrible. It isn't hard to write your own, if you need
    to, using Net::HTTP
    (http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html) and URI
    (http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html).

    -Justin
    Justin Collins, Jan 31, 2006
    #3
  4. On Jan 31, 2006, at 1:11 AM, Kev wrote:

    > Hi, I am looking for web crawler(spider) written in Ruby. I googled
    > but
    > couldn't find much information. Can anyone give me some links?


    The code I translated from Higher-Order Perl at the tail end of the
    following blog article may be of use to you.

    http://blog.grayproductions.net/articles/2006/01/31/iterators-
    chapters-4-and-5

    James Edward Gray II
    James Edward Gray II, Jan 31, 2006
    #4
  5. Kev

    Kev Guest

    ------=_Part_4753_29805324.1138795303153
    Content-Type: text/plain; charset=ISO-8859-1
    Content-Transfer-Encoding: quoted-printable
    Content-Disposition: inline

    Many thanks, I appreciate it very much.

    ------=_Part_4753_29805324.1138795303153--
    Kev, Feb 1, 2006
    #5
  6. Kev

    Bill Guindon Guest

    On 1/31/06, Kev <> wrote:
    > Hi, I am looking for web crawler(spider) written in Ruby. I googled but
    > couldn't find much information. Can anyone give me some links?


    Some more offerings here:
    http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/146169?146040=
    -146977

    now you're practically crawling with spiders :)

    > Thanks in advance.
    >
    > Kevin
    >
    >



    --
    Bill Guindon (aka aGorilla)
    The best answer to most questions is "it depends".
    Bill Guindon, Feb 1, 2006
    #6
  7. Kev

    James Britt Guest

    Gene Tani wrote:
    > Kev wrote:
    >
    >>Hi, I am looking for web crawler(spider) written in Ruby. I googled but
    >>couldn't find much information. Can anyone give me some links?
    >>

    >
    >
    > http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html


    There's not much spidering going on there, just some page scraping

    >
    > http://rubyforge.org/projects/wee/
    > look at Mechanize
    >


    My Mechanize example may be helpful, though.

    http://neurogami.com/cafe-fetcher/



    --
    James Britt

    http://www.ruby-doc.org - Ruby Help & Documentation
    http://www.artima.com/rubycs/ - The Journal By & For Rubyists
    http://www.rubystuff.com - The Ruby Store for Ruby Stuff
    http://www.jamesbritt.com - Playing with Better Toys
    http://www.30secondrule.com - Building Better Tools
    James Britt, Feb 2, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steve C. Orr, MCSD
    Replies:
    1
    Views:
    542
    reaway lee
    Aug 24, 2003
  2. Paul Morrison

    Web Crawler

    Paul Morrison, Oct 17, 2005, in forum: Java
    Replies:
    3
    Views:
    4,891
    lamantpirate
    Jun 30, 2012
  3. Sanjay Patra

    Web Crawler

    Sanjay Patra, Nov 17, 2004, in forum: C++
    Replies:
    2
    Views:
    680
  4. pm
    Replies:
    6
    Views:
    1,721
    Arne Vajhøj
    Jul 21, 2011
  5. KaiWen
    Replies:
    102
    Views:
    2,618
    Jorgen Grahn
    Sep 15, 2011
Loading...

Share This Page