Web spider

Discussion in 'Ruby' started by Lucas Carlson, May 4, 2005.

  1. Does anyone know of a web spider built in Ruby?
    Lucas Carlson, May 4, 2005
    #1
    1. Advertising

  2. Lucas Carlson

    leon breedt Guest

    On 5/4/05, Lucas Carlson <> wrote:
    > Does anyone know of a web spider built in Ruby?

    I couldn't find one either, so I wrote one based on http-access2 &
    htmltokenizer (on RAA). Its really domain-specific though, I'm not
    sure the code would be that useful to you.

    I could send you the code if you want to get an idea of whats
    involved, but be warned its really ugly (needed it fast).

    Leon
    leon breedt, May 4, 2005
    #2
    1. Advertising

  3. Lucas Carlson

    Mark Hubbart Guest

    On 5/4/05, Lucas Carlson <> wrote:
    > Does anyone know of a web spider built in Ruby?


    Here's an interesting link:
    http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf

    It's not a full-blown lib, but seems to be a pretty good tutorial for
    it. Regular expressions and simple http fetch methods make this sort
    of thing fairly easy to do in Ruby.
    Mark Hubbart, May 4, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. baroque Chou

    how google spider access my web site?

    baroque Chou, Jan 26, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    3,875
    Alan Silver
    Feb 2, 2006
  2. JeepGary
    Replies:
    2
    Views:
    462
    Roedy Green
    Oct 21, 2003
  3. Thomas Lindgaard

    Web Spider

    Thomas Lindgaard, Jul 6, 2004, in forum: Python
    Replies:
    3
    Views:
    584
    Peter Hansen
    Jul 7, 2004
  4. jdonnell
    Replies:
    5
    Views:
    550
    Peter Hansen
    Feb 17, 2005
  5. Replies:
    0
    Views:
    378
Loading...

Share This Page