Suggestions on a Web Spider that ....

Discussion in 'Java' started by JeepGary, Oct 21, 2003.

  1. JeepGary

    JeepGary Guest

    Hello,

    I am looking for a web spider that downloads the text content of a web
    page. Any ideas if one already exist. That would save a lot of time.

    Thanks,
    Gary
    JeepGary, Oct 21, 2003
    #1
    1. Advertising

  2. JeepGary

    Bjoern Guest

    JeepGary wrote:
    > Hello,
    >
    > I am looking for a web spider that downloads the text content of a web
    > page. Any ideas if one already exist. That would save a lot of time.


    I've had good results with Pavuk, http://www.idata.sk/~ondrej/pavuk/

    Worked better for me than wget.

    Bjoern

    --


    Let's not weep for their evil deeds,
    but for their lack of imagination
    (Nick Cave)
    Bjoern, Oct 21, 2003
    #2
    1. Advertising

  3. JeepGary

    Roedy Green Guest

    On 20 Oct 2003 18:36:25 -0700, (JeepGary) wrote or
    quoted :

    >I am looking for a web spider that downloads the text content of a web
    >page. Any ideas if one already exist. That would save a lot of time.


    For link checking, look at xenu. It is much faster than most since it
    is many thread. See http://mindprod.com/jgloss/xenu.html

    For code to programmatically download a page with an http GET see
    http://mindprod.com/fileio.html

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
    Roedy Green, Oct 21, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. baroque Chou

    how google spider access my web site?

    baroque Chou, Jan 26, 2006, in forum: ASP .Net
    Replies:
    7
    Views:
    3,882
    Alan Silver
    Feb 2, 2006
  2. Thomas Lindgaard

    Web Spider

    Thomas Lindgaard, Jul 6, 2004, in forum: Python
    Replies:
    3
    Views:
    584
    Peter Hansen
    Jul 7, 2004
  3. jdonnell
    Replies:
    5
    Views:
    554
    Peter Hansen
    Feb 17, 2005
  4. Victor

    ajax menu and web spider problem

    Victor, Apr 13, 2007, in forum: ASP .Net
    Replies:
    3
    Views:
    379
    Thomas Hansen
    Apr 18, 2007
  5. Replies:
    0
    Views:
    380
Loading...

Share This Page