Web spider

Discussion in 'Ruby' started by Lucas Carlson, May 4, 2005.

  1. Does anyone know of a web spider built in Ruby?
     
    Lucas Carlson, May 4, 2005
    #1
    1. Advertising

  2. Lucas Carlson

    leon breedt Guest

    On 5/4/05, Lucas Carlson <> wrote:
    > Does anyone know of a web spider built in Ruby?

    I couldn't find one either, so I wrote one based on http-access2 &
    htmltokenizer (on RAA). Its really domain-specific though, I'm not
    sure the code would be that useful to you.

    I could send you the code if you want to get an idea of whats
    involved, but be warned its really ugly (needed it fast).

    Leon
     
    leon breedt, May 4, 2005
    #2
    1. Advertising

  3. Lucas Carlson

    Mark Hubbart Guest

    On 5/4/05, Lucas Carlson <> wrote:
    > Does anyone know of a web spider built in Ruby?


    Here's an interesting link:
    http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf

    It's not a full-blown lib, but seems to be a pretty good tutorial for
    it. Regular expressions and simple http fetch methods make this sort
    of thing fairly easy to do in Ruby.
     
    Mark Hubbart, May 4, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. baroque Chou

    how google spider access my web site?

    baroque Chou, Jan 26, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    3,935
    Alan Silver
    Feb 2, 2006
  2. JeepGary
    Replies:
    2
    Views:
    486
    Roedy Green
    Oct 21, 2003
  3. Thomas Lindgaard

    Web Spider

    Thomas Lindgaard, Jul 6, 2004, in forum: Python
    Replies:
    3
    Views:
    613
    Peter Hansen
    Jul 7, 2004
  4. jdonnell
    Replies:
    5
    Views:
    582
    Peter Hansen
    Feb 17, 2005
  5. Replies:
    0
    Views:
    407
Loading...

Share This Page