Newbie - How do you request a web page from within Java?

[Jacky]

Hi,

How do you request a web page from within Java?

Regards,
Jacky

 

[Ryan]

Hi,

How do you request a web page from within Java?

Regards,
Jacky

Look at

http://java.sun.com/j2se/1.4.2/docs/api/java/net/HttpURLConnection.html

 

[jacky.newsgroup]

thanks!

 

[jacky.newsgroup]

Hi,

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

Thanks,
Jacky

 

[Roland]

Hi,

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

Thanks,
Jacky

<http://java.sun.com/docs/books/tutorial/networking/urls/readingURL.html>
<http://java.sun.com/docs/books/tutorial/networking/urls/index.html>
<http://java.sun.com/docs/books/tutorial/networking/index.html>
<http://java.sun.com/docs/books/tutorial/>
--
Regards,

Roland de Ruiter
___ ___
/__/ w_/ /__/
/ \ /_/ / \

 

[Wendy S]

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

There's also Jakarta Commons HTTPClient, which hides some of the complexity.
http://jakarta.apache.org/commons/httpclient/

 

[Dag Sunde]

Hi,

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

try {
URL url = new URL(http://www.google.com);
BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream()));
String str;
StringBuffer page = new StringBuffer();
while ((str = in.readLine()) != null) {
page.append(str + "\n");
}
in.close();
}
catch (MalformedURLException e) {
//...
}
catch (IOException e) {
//...
}

// And now 'page' contains the webpage...

 

[Ross Bamford]

Hi,

It seemed a bit complicated. Is there a source code sample, that
requests "http://www.google.com" for example?

Thanks,
Jacky

If you know you're going to get textual data back, something like the
following might work:

StringBuffer bigBuf = new StringBuffer();
try {
InputStream strm = new URL("http://www.google.com/").openStream();
BufferedReader rdr = new BufferedReader(new InputStreamReader(strm));

String thisLine = rdr.readLine();
while (thisLine != null) {
bigBuf = bigBuf.append(thisLine);
thisLine = rdr.readLine();
}
rdr.close();
} catch (Exception e) {
/* You might catch URLException, network exception, or IOException
}
String wholeThing = bigBuf.toString();

After which the 'wholeThing' string would contain all the data, unless
an exception occured.

You could easily modify this to check the URL's Content type - Read up
on the classes others have suggested. This is just a starter

Hope this helps,
Ross

 

[Jacky]

Thanks guys!

 

[Jacky]

Hi Guys,

I have the following code.
It displays alright when URL = http://google.com

It gives me an error when URL =
http://www.google.com/search?hl=en&...es+with+content+on+queer+theory,+gender&meta=

===============================================================
Error Message:
===============================================================
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL: htt
p://www.google.com/search?hl=en&safe=off&q=Media+studies+and+gender+studies+with+content+on+queer+th
eory%2C+gender&meta=
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:789)
at java.net.URL.openStream(URL.java:913)
at Project.main(Project.java:19)
Press any key to continue...



===============================================================
Code:
===============================================================
import java.io.*;
import java.net.*;
import java.lang.*;


public class Project {
public static void main(String[] args) throws Exception {
URL search_term = new URL("http://www.yahoo.com/");
BufferedReader in = new BufferedReader(
new InputStreamReader(
search_term.openStream()));

String inputLine;
StringBuffer bigBuf = new StringBuffer();


search_term = new URL("http://www.google.com/search?hl=en&...es+with+content+on+queer+theory,+gender&meta=");
in = new BufferedReader(new
InputStreamReader(search_term.openStream()));


while ((inputLine = in.readLine()) != null) {
bigBuf = bigBuf.append(inputLine);
bigBuf = bigBuf.append("\n\n\n");
}


String wholeThing = bigBuf.toString();


System.out.println(wholeThing);

in.close();
}
}

 

[jacky.newsgroup]

okay, I've checked out the Error code, google.com is forbidding me to
use Java program to use it's search function.

Any ways to get around it?

 

[Real Gagnon]

okay, I've checked out the Error code, google.com is forbidding me to

use Java program to use it's search function.

Any ways to get around it?

You need to pretend to be a "real" browser by setting a request property.

See http://www.rgagnon.com/javadetails/java-0399.html
for an example.

bye.

 

[jacky.newsgroup]

thanks!

btw, the code at the page has some "bugs":
webData = new BufferedReader(new
InputStreamReader(connection.getInputStream()));

connection should be replaced with con, because:
URLConnection con = urlObject.openConnection();
was declared eariler...


Here's the code:
=====================================================
import java.io.*;
import java.net.*;
import java.lang.*;


public class Project {
public static void main(String[] args) throws Exception {

String search= "dogs";
String google="http://www.google.com/search?q=" + search +
"&hl=en&ie=UTF-8&oe=UTF8";

URL urlObject = new URL(google);
URLConnection con = urlObject.openConnection();
con.setRequestProperty
( "User-Agent", "Mozilla/4.0 (compatible;MSIE 5.5; Windows NT 5.0;
H010818)" );

BufferedReader in = new BufferedReader(new
InputStreamReader(con.getInputStream()));




String inputLine;
StringBuffer bigBuf = new StringBuffer();

while ((inputLine = in.readLine()) != null) {
bigBuf = bigBuf.append(inputLine);
bigBuf = bigBuf.append("\n\n\n");
}


String wholeThing = bigBuf.toString();
System.out.println(wholeThing);

in.close();
}
}
=====================================================