Best references for buidling a webcrawler

[Jay Liu]

Hey guys,

I'm somewhat new to Java and I'm just looking for some reference ideas for
building a html page getter.
Basically a Wget in java. Can someone point in the right direction?

Thanks, Jay.

 

[Paul Lutus]

Hey guys,

I'm somewhat new to Java and I'm just looking for some reference ideas for
building a html page getter.
Basically a Wget in java. Can someone point in the right direction?

Start by experimenting with class URLConnection and look at the many online
examples.

 

[Alex Hunsley]

Hey guys,

I'm somewhat new to Java and I'm just looking for some reference ideas for
building a html page getter.
Basically a Wget in java. Can someone point in the right direction?

Thanks, Jay.

Hi Jay
You could look at URLConnection as Paul mentioned. I'd just like to
throw in my hat here and say have a look at HTTPClient too (part of the
Jakarta Commons project).

There's a comparison of the two here:

http://www.innovation.ch/java/HTTPClient/urlcon_vs_httpclient.html

(generally, HTTPClient does more things that URLConnection and is better
behaved)

You can visit the home of HTTPClient here:

http://jakarta.apache.org/commons/httpclient/


The Jakarta Commons project is good stuff. They have lots of other bits
and pieces that are very useful:

http://jakarta.apache.org/commons/


alex

 

[Tris Orendorff]

Hey guys,

I'm somewhat new to Java and I'm just looking for some reference ideas
for building a html page getter.

How about this, <http://www.jeffheaton.com/javabot/javaspider.shtml>, a simple spider program.


--
Sincerely,

Tris Orendorff

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d++ s+:- a+ C+ UL++++ P+ L+ E- W+ N++ o- K++ w+ O+ M !V PS+ PE Y+ PGP t+ !5 X- R- tv--- b++
DI++ D+ G++ e++ h---- r+++ y+++
------END GEEK CODE BLOCK------