Web crawler on python

Discussion in 'Python' started by sonich, Oct 26, 2008.

  1. sonich

    sonich Guest

    I need simple web crawler,
    I found Ruya, but it's seems not currently maintained.
    Does anybody know good web crawler on python or with python interface?
     
    sonich, Oct 26, 2008
    #1
    1. Advertising

  2. sonich

    Mr.SpOOn Guest

    On Sun, Oct 26, 2008 at 9:54 PM, sonich <> wrote:
    > I need simple web crawler,
    > I found Ruya, but it's seems not currently maintained.
    > Does anybody know good web crawler on python or with python interface?


    What about BeautifulSoup?

    http://www.crummy.com/software/BeautifulSoup/
     
    Mr.SpOOn, Oct 26, 2008
    #2
    1. Advertising

  3. sonich

    James Mills Guest

    On Mon, Oct 27, 2008 at 6:54 AM, sonich <> wrote:
    > I need simple web crawler,
    > I found Ruya, but it's seems not currently maintained.
    > Does anybody know good web crawler on python or with python interface?


    Simple, but it works. Extend it all you like.

    http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/examples/spider.py

    $ spider.py --help
    Usage: spider.py [options] <url>

    Options:
    --version show program's version number and exit
    -h, --help show this help message and exit
    -q, --quiet Enable quiet mode
    -l, --links Get links for specified url only
    -d DEPTH, --depth=DEPTH
    Maximum depth to traverse

    cheers
    James

    --
    --
    -- "Problems are solved by method"
     
    James Mills, Oct 26, 2008
    #3
  4. sonich

    Support Desk Guest

    -----Original Message-----
    From: James Mills [mailto:p]
    Sent: Sunday, October 26, 2008 5:26 PM
    To: sonich
    Cc:
    Subject: Re: Web crawler on python

    On Mon, Oct 27, 2008 at 6:54 AM, sonich <> wrote:
    > I need simple web crawler,
    > I found Ruya, but it's seems not currently maintained.
    > Does anybody know good web crawler on python or with python interface?


    Simple, but it works. Extend it all you like.

    http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e
    xamples/spider.py

    $ spider.py --help
    Usage: spider.py [options] <url>

    Options:
    --version show program's version number and exit
    -h, --help show this help message and exit
    -q, --quiet Enable quiet mode
    -l, --links Get links for specified url only
    -d DEPTH, --depth=DEPTH
    Maximum depth to traverse

    cheers
    James

    --
    --
    -- "Problems are solved by method"
     
    Support Desk, Oct 27, 2008
    #4
  5. sonich

    Alex Guest

    Alex, Oct 28, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. abhinav

    web crawler in python or C?

    abhinav, Feb 16, 2006, in forum: Python
    Replies:
    13
    Views:
    1,292
  2. abhinav

    web crawler in python or C?

    abhinav, Feb 16, 2006, in forum: C Programming
    Replies:
    1
    Views:
    1,427
  3. Replies:
    11
    Views:
    2,861
    subeen
    Jun 22, 2008
  4. yura

    Web crawler on python

    yura, Oct 30, 2008, in forum: Python
    Replies:
    1
    Views:
    326
    James Mills
    Oct 30, 2008
  5. Philip Semanchuk

    Re: web crawler in python

    Philip Semanchuk, Dec 10, 2009, in forum: Python
    Replies:
    0
    Views:
    493
    Philip Semanchuk
    Dec 10, 2009
Loading...

Share This Page