Forming HTTP request from java code

[Aditya Kumar]

Hi all,

I want to know, how should I go about forming an HTTP request from
(core) Java code. Is it possible?

I'd like to tell you the exact scenario I am dealing with here. I need
to send a search request (preferably to Google) from within the java
application and then show the response (search results), in plain
text, to the user using the application. I hope this gives a good
enough picture of what I am trying to achieve.

Since I have not written any code as of now, I can only share what I
have in my mind!

Any pointers/references will be appreciated. If there is anything else
you may need to know, please feel free to ask me.

Thanks in advance,

 

[Ian Wilson]

Hi all,

I want to know, how should I go about forming an HTTP request from
(core) Java code. Is it possible?

I'd like to tell you the exact scenario I am dealing with here. I need
to send a search request (preferably to Google) from within the java
application and then show the response (search results), in plain
text, to the user using the application. I hope this gives a good
enough picture of what I am trying to achieve.

Since I have not written any code as of now, I can only share what I
have in my mind!

Any pointers/references will be appreciated. If there is anything else
you may need to know, please feel free to ask me.

<http://www.cs.usfca.edu/~wolber/SoftwareDev/Distributed/WebServices/Google/googleJavaTutorial.htm>

 

[Aditya Kumar]

<http://www.cs.usfca.edu/~wolber/SoftwareDev/Distributed/WebServices/G...>- Hide quoted text -

- Show quoted text -

Ian,

I forgot to mention in my first message that while I was looking
around for a solution to my problem, it did occur to me that this
might lead me to playing with the Google API. Just that I was not sure
how to go about it and honestly, had not dug deep enough in the Google
API references/examples.

Thanks for the tip, will check it out.

Ciao,

 

[Ian Wilson]

Ian,

I forgot to mention in my first message that while I was looking
around for a solution to my problem, it did occur to me that this
might lead me to playing with the Google API. Just that I was not sure
how to go about it and honestly, had not dug deep enough in the Google
API references/examples.

Thanks for the tip, will check it out.

You may have been looking for Java class HTTPUrlConnection.

However Google specifically has terms and conditions that appear to
prohibit sending regular HTTP search queries from programs other than
interactive web-browsers.

I just found out that Google stopped issuing keys to their SOAP API a
while back. I suspect there's no legal way to write a Google-searching
app which isn't an interactive web browser.

 

[Gordon Beaton]

I suspect there's no legal way to write a Google-searching app which
isn't an interactive web browser.

When did TOC become "the law"?

/gordon

--

 

[Gordon Beaton]

When did TOC become "the law"?

Sorry that turned out to be a strange combination of ToS (Terms of
Service) and TaC (Terms and Conditions). I still wonder though.

/gordon

--

 

[Ian Wilson]

I think it was the 6th century BC, at least in a significant part of the
"civilized world" of that time. Probably millenia earlier, but I'm
neither lawyer nor historian

 

[Gordon Beaton]

I think it was the 6th century BC, at least in a significant part of
the "civilized world" of that time. Probably millenia earlier, but
I'm neither lawyer nor historian

I don't disagree that conditions agreed upon in a valid contract are
enforceable by law, however I don't think the situation referred to is
quite as simple as that.

In particular, I don't agree that it's even remotely illegal to
*write* a tool to browse Google, it's only some kinds of use they
prohibit.

Here are some random thoughts from a non-lawyer:

- If I use an automated tool to browse a website (not just Google), I
do not necessarily ever click "accept" or even see any contract. At
what point have I accepted any terms?

- In some jurisdictions (e.g. here in Sweden), shrink-wrap licenses
are not binding. I believe the same is true of click-through, but
I'm less sure about that.

- Not all terms are enforecable or even valid, regardless of what the
Big Company would have you believe or what you think you've agreed
to.

- The difference between an "interactive web browser" and a tool that
can do some additional processing of the information it reads is
semantic at best. At what point is my tool no longer an interactive
browser? Common browsers can do things like pre-fetching without me
knowing about it, and often follow links I never click on.

- wget and curl are two popular tools for the express purpose of
automating web queries. Have the authors of those programs broken
the law?

- What about web crawlers and spiders? Is robots.txt legally binding?

- I can't even find Google's ToS for its search engine.

/gordon

--

 

[Sanjay]

Hi all,

I want to know, how should I go about forming an HTTP request from
(core) Java code. Is it possible?

I'd like to tell you the exact scenario I am dealing with here. I need
to send a search request (preferably to Google) from within the java
application and then show the response (search results), in plain
text, to the user using the application. I hope this gives a good
enough picture of what I am trying to achieve.

Since I have not written any code as of now, I can only share what I
have in my mind!

Any pointers/references will be appreciated. If there is anything else
you may need to know, please feel free to ask me.

Thanks in advance,

Google has a WS API. Have you considered that?

 

[Ian Wilson]

- I can't even find Google's ToS for its search engine.

I Googled for "Google Terms and Conditions" and it located a document
that mentions Google's web sites and how one can agree to their T&Cs by
using their service.

Whether it's the one you seek or whether it is legally enforcable
anywhere are unknown to me.

Yahoo and MSN seem to have Java accessible APIs of the sort that Google
have ceased making available to new people. I bet they have T&Cs and
retain lawyers too

 

[Aditya Kumar]

You may have been looking for Java class HTTPUrlConnection.

Yes. And the good thing about it would be that since no "third-party"
API would be involved, the code shall remain "pure-Core" Java. I'd
like that. Funnily maybe, HTTPUrlConnection got lost in the
discussion! No one mentioned it!

 

[Aditya Kumar]

Google has a WS API. Have you considered that?

....Not anymore. It did exist but not anymore. The Google AJAX Search
API has seemingly replaced the WS API. It seems Google is discouraging
search requests generating from Non-Browser applications. The AJAX API
introduces itself as "the easiest way to incorporate a Google search
box on your site". So its obvious to me that they don't want anything
other than a web browser to make a request for a search.

I don't know, I may be wrong or missing something outright. Comments,
anyone?

 

[Sanjay]

...Not anymore. It did exist but not anymore. The Google AJAX Search
API has seemingly replaced the WS API. It seems Google is discouraging
search requests generating from Non-Browser applications. The AJAX API
introduces itself as "the easiest way to incorporate a Google search
box on your site". So its obvious to me that they don't want anything
other than a web browser to make a request for a search.

I don't know, I may be wrong or missing something outright. Comments,
anyone?

Yes, you are right. I used it once last year and not after that.

 

[Sanjay]

On the second thought, you can use Yahoo search, they provide SOAP API.
You can also use Windows Live search, they also provide a SOAP API. Of
course if that is an option.

 

[Pau]

Yes. And the good thing about it would be that since no "third-party"

API would be involved, the code shall remain "pure-Core" Java. I'd
like that. Funnily maybe, HTTPUrlConnection got lost in the
discussion! No one mentioned it!

You might want to check Fravia's page on bots: http://www.searchlores.org/bots.htm
Specially this simple java bot by Dolmen: http://www.searchlores.org/dolmen_2.htm

I am able to retrieve google homepage (and other pages too using the
bot), however I received an error when tried retrieving
http://www.google.co.in/search?q=bots.
You might want to play with the code.

Bipin Upadhyay.

 

[Bent C Dalager]

When did TOC become "the law"?

It's probably more a case of Google feeling free to block your ip if
they detect that you are behaving in a manner of which they do not
approve.

Also note that the concepts of "legal" and "illegal" are used in odd
ways in the field of computing. To wit:
java.lang.IllegalAccessError
java.lang.IllegalArgumentException
javax.crypto.IllegalBlockSizeException
etc.

Of course, Sun doesn't write the Law either and I don't think any of
those are going to land you in jail any time soon

Cheers
Bent D

 

[Gordon Beaton]

It's probably more a case of Google feeling free to block your ip if
they detect that you are behaving in a manner of which they do not
approve.

That was more or less the point I was trying to make.

/gordon

--

 

[Matej Cepl]

As a former lawyer, let me chime in with one terminological
comment -- at least in the American usage, law includes in the
particular relationship all private agreements between such
parties, meaning, that it is against the law to break a contract.
Of course, it doesn't mean that it is criminal offense or
something like that, determining appropriate remedy is just
a next step after recognition of illegal behavior, but breaking
Terms of Contract (is it what TOC stands for?) is against the
law.

Matej

 

[Lew]

As a former lawyer, let me chime in with one terminological
comment -- at least in the American usage, law includes in the
particular relationship all private agreements between such
parties, meaning, that it is against the law to break a contract.
Of course, it doesn't mean that it is criminal offense or
something like that, determining appropriate remedy is just
a next step after recognition of illegal behavior, but breaking
Terms of Contract (is it what TOC stands for?) is against the
law.

As explained upthread, the poster mangled two different acronyms together,
"T&C" (a.k.a. "TAC"), "Terms and Conditions", and "TOS", "Terms of Service".

At least in American usage, "agreement" requires all parties to agree. No
agreement, no contract. It was also mentioned upthread that users of bots
might not have even seen the TOS, much less agreed to them. Also mentioned
was that click-through "agreements", like shrink-wrap ones, do not allow the
user to contemplate and agree to the terms and therefore perhaps cannot
constitute a contract or any other binding obligations.

Also mentioned upthread is that the law is moot, since Google owns their own
servers and they don't have to let you play with them. Thus they are their
own law within their world, masters of their own domain.

Personally I think Google is very stuck up and full of themselves.