URLConnection tricks

Discussion in 'Java' started by Roedy Green, May 4, 2004.

  1. Roedy Green

    Roedy Green Guest

    URLConnection urlc = url.openConnection();
    urlc.setAllowUserInteraction( false );
    urlc.setDoInput( true );
    urlc.setDoOutput( false );
    urlc.setUseCaches( true );
    urlc.connect();
    String mime = urlc.getContentType();


    How would I modify that code to avoid getting the contents of the
    file, get just the headers?

    How would I turn off Keep-Alive?

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 4, 2004
    #1
    1. Advertising

  2. Roedy Green schrieb:

    > URLConnection urlc = url.openConnection();
    > urlc.setAllowUserInteraction( false );
    > urlc.setDoInput( true );
    > urlc.setDoOutput( false );
    > urlc.setUseCaches( true );
    > urlc.connect();
    > String mime = urlc.getContentType();
    >
    >
    >How would I modify that code to avoid getting the contents of the
    >file, get just the headers?
    >

    don't know...

    >
    >How would I turn off Keep-Alive?
    >

    I think
    urlc.setRequestProperty("Connection", "Close");
    should do the trick. See http://www.faqs.org/rfcs/rfc2068.html (chapter
    "14.10 Connection").
    But I didn't actually test it.

    --
    Thomas<dot>Fritsch<squiggle>ops<dot>de
     
    Thomas Fritsch, May 4, 2004
    #2
    1. Advertising

  3. On Tue, 04 May 2004 05:30:03 GMT, Roedy Green wrote:
    > URLConnection urlc = url.openConnection();
    > urlc.setAllowUserInteraction( false );
    > urlc.setDoInput( true );
    > urlc.setDoOutput( false );
    > urlc.setUseCaches( true );
    > urlc.connect();
    > String mime = urlc.getContentType();
    >
    >
    > How would I modify that code to avoid getting the contents of the
    > file, get just the headers?


    Use HEAD instead of GET. They are identical, except that the server
    response must not include a message body. That probably rules out the
    use of URLConnection though.

    > How would I turn off Keep-Alive?


    Include the request header "Connection: close".

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
     
    Gordon Beaton, May 4, 2004
    #3
  4. Roedy Green

    Chris Uppal Guest

    Roedy Green wrote:
    > URLConnection urlc = url.openConnection();
    > urlc.setAllowUserInteraction( false );
    > urlc.setDoInput( true );
    > urlc.setDoOutput( false );
    > urlc.setUseCaches( true );
    > urlc.connect();
    > String mime = urlc.getContentType();
    >
    >
    > How would I modify that code to avoid getting the contents of the
    > file, get just the headers?


    IIRC: cast the URLConnection to an HttpURLConnection and use
    setRequestMethod("HEAD") before you connect().

    BTW, a surprisingly large number of servers ignore the "HEAD" and treat it as a
    "GET" -- that shouldn't affect your logic, but it can be confusing if you are
    debugging with a network sniffer.

    -- chris
     
    Chris Uppal, May 4, 2004
    #4
  5. Roedy Green schrieb:

    > URLConnection urlc = url.openConnection();
    > urlc.setAllowUserInteraction( false );
    > urlc.setDoInput( true );
    > urlc.setDoOutput( false );
    > urlc.setUseCaches( true );
    > urlc.connect();
    > String mime = urlc.getContentType();
    >
    >
    >How would I modify that code to avoid getting the contents of the
    >file, get just the headers?
    >

    See
    http://java.sun.com/j2se/1.4.2/docs...ection.html#setRequestMethod(java.lang.String)
    ==> Probably
    ((HttpURLConnection) urlc).setRequestMethod("HEAD");
    will do. But obviously this depends on my unproved assumption, that
    url.openConnection()
    returns an instance of java.net.HttpURLConnection.

    >
    >How would I turn off Keep-Alive?
    >
    >
    >
    >

    --
    Thomas<dot>Fritsch<squiggle>ops<dot>de
     
    Thomas Fritsch, May 4, 2004
    #5
  6. Roedy Green

    Dave Monroe Guest

    Roedy Green <> wrote in message news:<>...
    > URLConnection urlc = url.openConnection();
    > urlc.setAllowUserInteraction( false );
    > urlc.setDoInput( true );
    > urlc.setDoOutput( false );
    > urlc.setUseCaches( true );
    > urlc.connect();
    > String mime = urlc.getContentType();
    >
    >
    > How would I modify that code to avoid getting the contents of the
    > file, get just the headers?
    >
    > How would I turn off Keep-Alive?



    Have a look at my implementation of wget. The HttpUrlConnection class
    inherits from URLConnection. The header info is visible there.

    I'm pretty sure Keep-Alive is a tcp function.

    import java.net.*;
    import java.io.*;
    import java.util.*;

    public class wget {
    public static void main(String [] args) {
    if(args.length == 0) {
    System.err.println("usage: java wget url");
    System.exit(1);
    }

    try {
    URL u = new URL(args[0]);
    HttpURLConnection h =
    (HttpURLConnection)u.openConnection();

    InputStream is = h.getInputStream();

    BufferedReader in = new BufferedReader(
    new InputStreamReader(is));

    String line;

    while ((line = in.readLine()) != null) {
    System.out.println(line);
    }

    }
    catch(Exception e) {
    System.err.println(e);
    System.exit(1);
    }
    }
    }
     
    Dave Monroe, May 4, 2004
    #6
  7. Dave Monroe wrote:

    > Have a look at my implementation of wget. The HttpUrlConnection class
    > inherits from URLConnection. The header info is visible there.


    Okay.

    > I'm pretty sure Keep-Alive is a tcp function.


    It is, but I suspect Roedy actually meant to ask about the "Persistent
    Connection" feature of HTTP 1.1. The HttpURLConnection API docs do say
    "Calling the disconnect() method may close the underlying socket if a
    persistent connection is otherwise idle at that time," which is at least
    one avenue of approach. If checking many URLs on the same server,
    however, then it may be counterproductive to do that.

    > public class wget {
    > public static void main(String [] args) {


    [...]

    > URL u = new URL(args[0]);
    > HttpURLConnection h =
    > (HttpURLConnection)u.openConnection();


    And that should work reliably, as long as the URL is an HTTP URL.

    > InputStream is = h.getInputStream();
    >
    > BufferedReader in = new BufferedReader(
    > new InputStreamReader(is));


    That, however, is pretty broken. It depends on the system's default
    charset being suitable. It will work for ASCII, UTF-8, or any of the
    ISO-8859 family of charsets, as they all agree on the first 128 code
    points, but it will break on many others.

    > String line;
    >
    > while ((line = in.readLine()) != null) {
    > System.out.println(line);
    > }


    And that's pretty naive. It does not account for any content coding
    that might have been specified in the header, nor (in the case of HTML
    content) for a charset specified via a meta tag. It also doesn't do
    anything sensible with content that is inherently binary.

    [...]

    That also doesn't answer the actual question, which was about obtaining
    the content type without retrieving the entire content. Assuming that
    the point is to reduce impact on the server, using the "HEAD" HTTP
    method to make the request is the only way that has a reasonable chance
    of solving the problem. If the concern is more for the client side then
    the client could always close the connection without reading the content
    -- for large entities that might prevent the content from being fully
    retrieved when the server treats "HEAD" as "GET".


    John Bollinger
     
    John C. Bollinger, May 4, 2004
    #7
  8. Roedy Green

    Roedy Green Guest

    On 4 May 2004 05:23:10 -0700, (Dave Monroe)
    wrote or quoted :

    >
    >I'm pretty sure Keep-Alive is a tcp function.


    Connect: Keep-Alive is also a field in the GET header to say, keep the
    socket open after sending the response for more traffic. Most of the
    time you don't need it and you needlessly keep the server dangling.


    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 4, 2004
    #8
  9. Roedy Green

    Roedy Green Guest

    On Tue, 04 May 2004 05:30:03 GMT, Roedy Green
    <> wrote or quoted :

    >How would I modify that code to avoid getting the contents of the
    >file, get just the headers?
    >
    >How would I turn off Keep-Alive?


    thanks everyone. MimeCheck 1.1 now uses this information and FileI/O
    4.5 teaches it to the newbies.

    see http://mindprod.com/jgloss/mime.html
    http://mindprod.com/fileio.html

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 4, 2004
    #9
  10. On Tue, 04 May 2004 17:19:57 GMT, Roedy Green wrote:

    > see http://mindprod.com/jgloss/mime.html


    Neat! Would you consider adding some JS(!)
    to allow configuring it by URL?

    If your applet accepted a 'url' parameter,
    you could write the parameter into the applet
    tag using JS, or default to the 'no param'
    applet for those UA's with no JS.

    Yes, yes. I am familiar with your aversion
    to mixing JS/Java. But you can gain extra
    functionality while losing nothing in this
    case.

    The ability to specify the entire check
    as an URL would be most handy.
    [ I can recall situations in the c.i.w.a.h.
    and c.i.w.a.s groups where I could show
    someone what is going wrong using the
    added functionality. ]

    --
    Andrew Thompson
    http://www.PhySci.org/ Open-source software suite
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.1point1C.org/ Science & Technology
     
    Andrew Thompson, May 4, 2004
    #10
  11. Roedy Green

    Roedy Green Guest

    On Tue, 04 May 2004 18:03:27 GMT, Andrew Thompson
    <> wrote or quoted :

    >Neat! Would you consider adding some JS(!)
    >to allow configuring it by URL?


    You want to invoke the applet, and give it a parameter with just an
    URL.

    e.g. http://mindprod.com/mimecheck.html?http:somewhere.org/app.jnlp

    JavaScript on that page takes the parameter, and feeds it to MimeCheck
    as an Applet tag, which then starts up and displays the URL, and
    auto-presses TEST. The browser then views the completed applet page.

    I don't know enough JavaScript to do that, but I do know how to add a
    tag called URL to the Applet. If you want to cook up the javascript
    part, MimeCheck version 1.2 now has an optional

    <param name="URL" value="http://mindprod.com/esper.jnlp">



    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 4, 2004
    #11
  12. On Tue, 04 May 2004 18:55:47 GMT, Roedy Green wrote:

    > On Tue, 04 May 2004 18:03:27 GMT, Andrew Thompson
    >>Neat! Would you consider adding some JS(!)
    >>to allow configuring it by URL?

    >
    > You want to invoke the applet, and give it a parameter with just an
    > URL.


    Yep.

    > If you want to cook up the javascript part,


    Done.

    > MimeCheck version 1.2 now has an optional
    >
    > <param name="URL" value="http://mindprod.com/esper.jnlp">


    Cool. Check your email.

    --
    Andrew Thompson
    http://www.PhySci.org/ Open-source software suite
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.1point1C.org/ Science & Technology
     
    Andrew Thompson, May 4, 2004
    #12
  13. Roedy Green

    Roedy Green Guest

    On Tue, 04 May 2004 20:01:57 GMT, Andrew Thompson
    <> wrote or quoted :

    >> <param name="URL" value="http://mindprod.com/esper.jnlp">

    >
    >Cool. Check your email.


    no sign of anything. try my first name, my last initial @ my website.
    e.g. http://mindprod.com/images/mailto.gif

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 5, 2004
    #13
  14. On Tue, 04 May 2004 23:26:51 GMT, Roedy Green wrote:

    > On Tue, 04 May 2004 20:01:57 GMT, Andrew Thompson

    ...
    >>...Check your email.

    >
    > no sign of anything. try my first name, my last initial @ my website.
    > e.g. http://mindprod.com/images/mailto.gif


    D'Oh! I was misspelling 'mindprod'.
    Let's try a different route..

    For the files..
    <http://www.physci.org/test/002mimetype/>

    And just to see it work..
    <http://www.physci.org/test/002mimetype/mimetype.html?url=http://www.physci.org/jnlp/PhySci.jnlp>

    [ I have also updated you address in my mail program ]

    --
    Andrew Thompson
    http://www.PhySci.org/ Open-source software suite
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.1point1C.org/ Science & Technology
     
    Andrew Thompson, May 5, 2004
    #14
  15. Andrew Thompson, May 5, 2004
    #15
  16. Roedy Green

    Roedy Green Guest

    On Wed, 05 May 2004 06:10:10 GMT, Andrew Thompson
    <> wrote or quoted :

    >
    >For the files..
    ><http://www.physci.org/test/002mimetype/>


    Your code is now hooked up. You can now test a url all on the command
    line.

    For example to test the MIME types being sent by
    http://java.sun.com/index.jsp you would type into your browser's
    command line:
    http://mindprod.com/urlcheck.html?url=http://java.sun.com/index.jsp

    or as a tinyurl:
    http://tinyurl.com/ysbrz
    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 5, 2004
    #16
  17. On Wed, 05 May 2004 21:29:51 GMT, Roedy Green wrote:

    >>For the files..

    ...
    > Your code is now hooked up. You can now test a url all on the command
    > line.


    Excellent, I checked it worked from here
    and removed the files from my server.

    > For example to test the MIME types being sent by
    > http://java.sun.com/index.jsp you would type into your browser's
    > command line:
    > http://mindprod.com/urlcheck.html?url=http://java.sun.com/index.jsp


    Not that you can _also_ use..

    http://mindprod.com/urlcheck.html?=http://java.sun.com/index.jsp
    or
    <http://mindprod.com/urlcheck.html?thisisaverylongandsillyparameternamethatthejavascriptwillutterlyignore=http://java.sun.com/index.jsp>

    ;-)

    (shrugs) The 'url' bit seemed logical,
    but the JS will work without it. [All
    it needs is '?=' before the actual URL ]

    --
    Andrew Thompson
    http://www.PhySci.org/ Open-source software suite
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.1point1C.org/ Science & Technology
     
    Andrew Thompson, May 6, 2004
    #17
  18. Roedy Green

    Roedy Green Guest

    On Thu, 06 May 2004 03:03:57 GMT, Andrew Thompson
    <> wrote or quoted :

    >Not that you can _also_ use..


    Not really. that is not the documented interface. I reserve the
    right to create new optional parameters, and if you used something
    other than url, it could stop working.
    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
     
    Roedy Green, May 6, 2004
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kristoffel

    URLConnection and proxy

    Kristoffel, Jul 27, 2003, in forum: Java
    Replies:
    0
    Views:
    1,590
    Kristoffel
    Jul 27, 2003
  2. Brock Jones
    Replies:
    0
    Views:
    2,656
    Brock Jones
    Sep 4, 2003
  3. Bronwyn Howard
    Replies:
    0
    Views:
    6,141
    Bronwyn Howard
    Oct 1, 2003
  4. dave
    Replies:
    0
    Views:
    865
  5. Replies:
    2
    Views:
    3,709
Loading...

Share This Page