Forming HTTP request from java code

Discussion in 'Java' started by Aditya Kumar, May 8, 2007.

  1. Aditya Kumar

    Aditya Kumar Guest

    Hi all,

    I want to know, how should I go about forming an HTTP request from
    (core) Java code. Is it possible?

    I'd like to tell you the exact scenario I am dealing with here. I need
    to send a search request (preferably to Google) from within the java
    application and then show the response (search results), in plain
    text, to the user using the application. I hope this gives a good
    enough picture of what I am trying to achieve.

    Since I have not written any code as of now, I can only share what I
    have in my mind!

    Any pointers/references will be appreciated. If there is anything else
    you may need to know, please feel free to ask me.

    Thanks in advance,
     
    Aditya Kumar, May 8, 2007
    #1
    1. Advertising

  2. Aditya Kumar

    Ian Wilson Guest

    Aditya Kumar wrote:
    > Hi all,
    >
    > I want to know, how should I go about forming an HTTP request from
    > (core) Java code. Is it possible?
    >
    > I'd like to tell you the exact scenario I am dealing with here. I need
    > to send a search request (preferably to Google) from within the java
    > application and then show the response (search results), in plain
    > text, to the user using the application. I hope this gives a good
    > enough picture of what I am trying to achieve.
    >
    > Since I have not written any code as of now, I can only share what I
    > have in my mind!
    >
    > Any pointers/references will be appreciated. If there is anything else
    > you may need to know, please feel free to ask me.
    >


    <http://www.cs.usfca.edu/~wolber/SoftwareDev/Distributed/WebServices/Google/googleJavaTutorial.htm>
     
    Ian Wilson, May 8, 2007
    #2
    1. Advertising

  3. Aditya Kumar

    Aditya Kumar Guest


    > <http://www.cs.usfca.edu/~wolber/SoftwareDev/Distributed/WebServices/G...>- Hide quoted text -
    >
    > - Show quoted text -


    Ian,

    I forgot to mention in my first message that while I was looking
    around for a solution to my problem, it did occur to me that this
    might lead me to playing with the Google API. Just that I was not sure
    how to go about it and honestly, had not dug deep enough in the Google
    API references/examples.

    Thanks for the tip, will check it out.

    Ciao,
     
    Aditya Kumar, May 8, 2007
    #3
  4. Aditya Kumar

    Ian Wilson Guest

    Aditya Kumar wrote:
    >><http://www.cs.usfca.edu/~wolber/SoftwareDev/Distributed/WebServices/G...>- Hide quoted text -
    >>
    >>- Show quoted text -

    >
    >
    > Ian,
    >
    > I forgot to mention in my first message that while I was looking
    > around for a solution to my problem, it did occur to me that this
    > might lead me to playing with the Google API. Just that I was not sure
    > how to go about it and honestly, had not dug deep enough in the Google
    > API references/examples.
    >
    > Thanks for the tip, will check it out.
    >


    You may have been looking for Java class HTTPUrlConnection.

    However Google specifically has terms and conditions that appear to
    prohibit sending regular HTTP search queries from programs other than
    interactive web-browsers.

    I just found out that Google stopped issuing keys to their SOAP API a
    while back. I suspect there's no legal way to write a Google-searching
    app which isn't an interactive web browser.
     
    Ian Wilson, May 8, 2007
    #4
  5. On Tue, 08 May 2007 11:46:20 +0100, Ian Wilson wrote:
    > I suspect there's no legal way to write a Google-searching app which
    > isn't an interactive web browser.


    When did TOC become "the law"?

    /gordon

    --
     
    Gordon Beaton, May 8, 2007
    #5
  6. On 08 May 2007 10:49:47 GMT, Gordon Beaton wrote:
    > On Tue, 08 May 2007 11:46:20 +0100, Ian Wilson wrote:
    >> I suspect there's no legal way to write a Google-searching app which
    >> isn't an interactive web browser.

    >
    > When did TOC become "the law"?


    Sorry that turned out to be a strange combination of ToS (Terms of
    Service) and TaC (Terms and Conditions). I still wonder though.

    /gordon

    --
     
    Gordon Beaton, May 8, 2007
    #6
  7. Aditya Kumar

    Ian Wilson Guest

    OT: When did contract conditions become enforcable in a court of

    Gordon Beaton wrote:
    > On 08 May 2007 10:49:47 GMT, Gordon Beaton wrote:
    >
    >>
    >> When did TOC become "the law"?


    I think it was the 6th century BC, at least in a significant part of the
    "civilized world" of that time. Probably millenia earlier, but I'm
    neither lawyer nor historian :)
     
    Ian Wilson, May 8, 2007
    #7
  8. Re: OT: When did contract conditions become enforcable in a court of law? Was Re: Forming HTTP request from java code

    On Tue, 08 May 2007 14:45:02 +0100, Ian Wilson wrote:
    > Gordon Beaton wrote:
    >> On 08 May 2007 10:49:47 GMT, Gordon Beaton wrote:
    >>
    >>>
    >>> When did TOC become "the law"?

    >
    > I think it was the 6th century BC, at least in a significant part of
    > the "civilized world" of that time. Probably millenia earlier, but
    > I'm neither lawyer nor historian :)


    I don't disagree that conditions agreed upon in a valid contract are
    enforceable by law, however I don't think the situation referred to is
    quite as simple as that.

    In particular, I don't agree that it's even remotely illegal to
    *write* a tool to browse Google, it's only some kinds of use they
    prohibit.

    Here are some random thoughts from a non-lawyer:

    - If I use an automated tool to browse a website (not just Google), I
    do not necessarily ever click "accept" or even see any contract. At
    what point have I accepted any terms?

    - In some jurisdictions (e.g. here in Sweden), shrink-wrap licenses
    are not binding. I believe the same is true of click-through, but
    I'm less sure about that.

    - Not all terms are enforecable or even valid, regardless of what the
    Big Company would have you believe or what you think you've agreed
    to.

    - The difference between an "interactive web browser" and a tool that
    can do some additional processing of the information it reads is
    semantic at best. At what point is my tool no longer an interactive
    browser? Common browsers can do things like pre-fetching without me
    knowing about it, and often follow links I never click on.

    - wget and curl are two popular tools for the express purpose of
    automating web queries. Have the authors of those programs broken
    the law?

    - What about web crawlers and spiders? Is robots.txt legally binding?

    - I can't even find Google's ToS for its search engine.

    /gordon

    --
     
    Gordon Beaton, May 8, 2007
    #8
  9. Aditya Kumar

    Sanjay Guest

    Aditya Kumar wrote:
    > Hi all,
    >
    > I want to know, how should I go about forming an HTTP request from
    > (core) Java code. Is it possible?
    >
    > I'd like to tell you the exact scenario I am dealing with here. I need
    > to send a search request (preferably to Google) from within the java
    > application and then show the response (search results), in plain
    > text, to the user using the application. I hope this gives a good
    > enough picture of what I am trying to achieve.
    >
    > Since I have not written any code as of now, I can only share what I
    > have in my mind!
    >
    > Any pointers/references will be appreciated. If there is anything else
    > you may need to know, please feel free to ask me.
    >
    > Thanks in advance,
    >

    Google has a WS API. Have you considered that?
     
    Sanjay, May 8, 2007
    #9
  10. Aditya Kumar

    Ian Wilson Guest

    Re: OT: When did contract conditions become enforcable in a court

    Gordon Beaton wrote:
    >
    > - I can't even find Google's ToS for its search engine.
    >


    I Googled for "Google Terms and Conditions" and it located a document
    that mentions Google's web sites and how one can agree to their T&Cs by
    using their service.

    Whether it's the one you seek or whether it is legally enforcable
    anywhere are unknown to me.

    Yahoo and MSN seem to have Java accessible APIs of the sort that Google
    have ceased making available to new people. I bet they have T&Cs and
    retain lawyers too :)
     
    Ian Wilson, May 8, 2007
    #10
  11. Aditya Kumar

    Aditya Kumar Guest


    > You may have been looking for Java class HTTPUrlConnection.


    Yes. And the good thing about it would be that since no "third-party"
    API would be involved, the code shall remain "pure-Core" Java. I'd
    like that. Funnily maybe, HTTPUrlConnection got lost in the
    discussion! No one mentioned it!
     
    Aditya Kumar, May 8, 2007
    #11
  12. Aditya Kumar

    Aditya Kumar Guest


    > Google has a WS API. Have you considered that?


    ....Not anymore. It did exist but not anymore. The Google AJAX Search
    API has seemingly replaced the WS API. It seems Google is discouraging
    search requests generating from Non-Browser applications. The AJAX API
    introduces itself as "the easiest way to incorporate a Google search
    box on your site". So its obvious to me that they don't want anything
    other than a web browser to make a request for a search.

    I don't know, I may be wrong or missing something outright. Comments,
    anyone?
     
    Aditya Kumar, May 8, 2007
    #12
  13. Aditya Kumar

    Sanjay Guest

    Aditya Kumar wrote:
    >> Google has a WS API. Have you considered that?

    >
    > ...Not anymore. It did exist but not anymore. The Google AJAX Search
    > API has seemingly replaced the WS API. It seems Google is discouraging
    > search requests generating from Non-Browser applications. The AJAX API
    > introduces itself as "the easiest way to incorporate a Google search
    > box on your site". So its obvious to me that they don't want anything
    > other than a web browser to make a request for a search.
    >
    > I don't know, I may be wrong or missing something outright. Comments,
    > anyone?
    >
    >


    Yes, you are right. I used it once last year and not after that.
     
    Sanjay, May 8, 2007
    #13
  14. Aditya Kumar

    Sanjay Guest

    On the second thought, you can use Yahoo search, they provide SOAP API.
    You can also use Windows Live search, they also provide a SOAP API. Of
    course if that is an option.
     
    Sanjay, May 8, 2007
    #14
  15. Aditya Kumar

    Pau Guest

    > Yes. And the good thing about it would be that since no "third-party"
    > API would be involved, the code shall remain "pure-Core" Java. I'd
    > like that. Funnily maybe, HTTPUrlConnection got lost in the
    > discussion! No one mentioned it!


    You might want to check Fravia's page on bots: http://www.searchlores.org/bots.htm
    Specially this simple java bot by Dolmen: http://www.searchlores.org/dolmen_2.htm

    I am able to retrieve google homepage (and other pages too using the
    bot), however I received an error when tried retrieving
    http://www.google.co.in/search?q=bots.
    You might want to play with the code.

    Bipin Upadhyay.
     
    Pau, May 9, 2007
    #15
  16. In article <464055cb$0$31529$>,
    Gordon Beaton <> wrote:
    >On Tue, 08 May 2007 11:46:20 +0100, Ian Wilson wrote:
    >> I suspect there's no legal way to write a Google-searching app which
    >> isn't an interactive web browser.

    >
    >When did TOC become "the law"?


    It's probably more a case of Google feeling free to block your ip if
    they detect that you are behaving in a manner of which they do not
    approve.

    Also note that the concepts of "legal" and "illegal" are used in odd
    ways in the field of computing. To wit:
    java.lang.IllegalAccessError
    java.lang.IllegalArgumentException
    javax.crypto.IllegalBlockSizeException
    etc.

    Of course, Sun doesn't write the Law either and I don't think any of
    those are going to land you in jail any time soon :)

    Cheers
    Bent D
    --
    Bent Dalager - - http://www.pvv.org/~bcd
    powered by emacs
     
    Bent C Dalager, May 9, 2007
    #16
  17. On Wed, 9 May 2007 10:28:10 +0000 (UTC), Bent C Dalager wrote:
    > It's probably more a case of Google feeling free to block your ip if
    > they detect that you are behaving in a manner of which they do not
    > approve.


    That was more or less the point I was trying to make.

    /gordon

    --
     
    Gordon Beaton, May 9, 2007
    #17
  18. Aditya Kumar

    Matej Cepl Guest

    On 2007-05-09, 10:28 GMT, Bent C Dalager wrote:
    > In article <464055cb$0$31529$>,
    > Gordon Beaton <> wrote:
    >>On Tue, 08 May 2007 11:46:20 +0100, Ian Wilson wrote:
    >>> I suspect there's no legal way to write a Google-searching app which
    >>> isn't an interactive web browser.

    >>
    >>When did TOC become "the law"?


    As a former lawyer, let me chime in with one terminological
    comment -- at least in the American usage, law includes in the
    particular relationship all private agreements between such
    parties, meaning, that it is against the law to break a contract.
    Of course, it doesn't mean that it is criminal offense or
    something like that, determining appropriate remedy is just
    a next step after recognition of illegal behavior, but breaking
    Terms of Contract (is it what TOC stands for?) is against the
    law.

    Matej
     
    Matej Cepl, May 9, 2007
    #18
  19. Aditya Kumar

    Lew Guest

    Matej Cepl wrote:
    > On 2007-05-09, 10:28 GMT, Bent C Dalager wrote:
    >> In article <464055cb$0$31529$>,
    >> Gordon Beaton <> wrote:
    >>> On Tue, 08 May 2007 11:46:20 +0100, Ian Wilson wrote:
    >>>> I suspect there's no legal way to write a Google-searching app which
    >>>> isn't an interactive web browser.
    >>> When did TOC become "the law"?

    >
    > As a former lawyer, let me chime in with one terminological
    > comment -- at least in the American usage, law includes in the
    > particular relationship all private agreements between such
    > parties, meaning, that it is against the law to break a contract.
    > Of course, it doesn't mean that it is criminal offense or
    > something like that, determining appropriate remedy is just
    > a next step after recognition of illegal behavior, but breaking
    > Terms of Contract (is it what TOC stands for?) is against the
    > law.


    As explained upthread, the poster mangled two different acronyms together,
    "T&C" (a.k.a. "TAC"), "Terms and Conditions", and "TOS", "Terms of Service".

    At least in American usage, "agreement" requires all parties to agree. No
    agreement, no contract. It was also mentioned upthread that users of bots
    might not have even seen the TOS, much less agreed to them. Also mentioned
    was that click-through "agreements", like shrink-wrap ones, do not allow the
    user to contemplate and agree to the terms and therefore perhaps cannot
    constitute a contract or any other binding obligations.

    Also mentioned upthread is that the law is moot, since Google owns their own
    servers and they don't have to let you play with them. Thus they are their
    own law within their world, masters of their own domain.

    Personally I think Google is very stuck up and full of themselves.

    --
    Lew
     
    Lew, May 9, 2007
    #19
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Henry Rawas [MSFT]
    Replies:
    0
    Views:
    1,078
    Henry Rawas [MSFT]
    Sep 24, 2004
  2. pramod
    Replies:
    3
    Views:
    305
    Olivier Dubuisson
    Dec 29, 2003
  3. Replies:
    3
    Views:
    289
    Fredrik Lundh
    Nov 12, 2005
  4. Replies:
    6
    Views:
    395
    Robbie Hatley
    Jul 14, 2006
  5. Jeff Rush
    Replies:
    0
    Views:
    245
    Jeff Rush
    Jul 24, 2007
Loading...

Share This Page