Connecting to dynamically generated data

[Derek]

Hello all, this is my first time posting to a Usenet group, so please
forgive me for my "green-ness." Anyway, my problem is this: I need to
connect to a website that provides current and historical weather data
and does so through a web interface. To get the information you pass it
the type of data you want and a few constraints and the server
retrieves the data in plain text.

Here's an example:
http://nomads.ncdc.noaa.gov:9091/do...6_0600_fff.asc?tmp[0:0][0:6:25][0:180][0:359]

However, if I try to connect to the page with the following,

URL url = new
URL("http://nomads.ncdc.noaa.gov:9091/do...0600_fff.ascii?tmp[0:0][0:6:25][0:180][0:359]");
InputStream stream = url.openStream();

I get the following Exception:
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown
Source)
at java.net.URL.openStream(Unknown Source)

Now, I've already set the proxy so that it can connect to outside web
pages (I've already tried it out on more mundane pages with no
problems), so I'm not sure what else I need to do to be able to connect
to it to retrieve the data. Can anyone help me out? Many thanks!

-Derek

 

[Derek]

Ugh, links got all screwed up. Try copying this:
nomads.ncdc.noaa.gov:9091/dods/NCEP_GFS/200506/20050616/gfs_3_20050616_0600_fff.asc?tmp[0:0][0:6:25][0:180][0:359]

 

[Andrew Thompson]

Hello all, this is my first time posting to a Usenet group, so please
forgive me for my "green-ness."

Cool. First tip. Please repeat the key words of the subject
in the body of the post, as some people ignore
subject lines..

Sub: Connecting to dynamically generated data

'Dynamically generated', eh..? That could be the problem
(though I'm not sure).

..Anyway, my problem is this: I need to
connect to a website

Which one?

http://nomads.ncdc.noaa.gov:9091/do...6_0600_fff.asc?tmp[0:0][0:6:25][0:180][0:359]

Oh right.. <http://www.noaa.gov/>
The weather folks, they have some interesting data.
They also seem relatively Java friendly..
<http://shovel.hpcc.noaa.gov/cgi-bin...and&format=builtin-long&sort=score&words=java>

Are you sure they do not already provide an API for providing
this data to Java processes?

However, if I try to connect to the page with the following,

URL url = new
URL("http://nomads.ncdc.noaa.gov:9091/do...0600_fff.ascii?tmp[0:0][0:6:25][0:180][0:359]");
InputStream stream = url.openStream();

I get the following Exception:
java.net.SocketException: Unexpected end of file from server

I did a quick check here to the point of opening an input stream,
and got no errors. Are you sure the error indicated that *exact* line?

...so I'm not sure what else I need to do to be able to connect
to it to retrieve the data.

I strongly recommend getting in touch with the noaa support, I
suspect that
a) the 'dynamic' nature of the information feed is the problem.
b) if they are OK with direct programmatic access, they will
either point you to the API designed to access it, or maybe point
you to the little FAQ that shows you the trick.

HTH

--
Andrew Thompson
physci.org 1point1c.org javasaver.com lensescapes.com athompson.info
"..I pick up all the pieces and make an island. Might even raise just a
little sand"
Jimi Hendrix 'Voodoo Chile (Slight Return)'

 

[Derek]

Thanks a bunch for the reply! I was hoping that because the data just
generates and presents itself if you try to access it through a web
browser that it would do the same when trying to connect to it through
Java. I'll try your suggestions, hopefully there is an API I can use.
Thanks again!

 

[Roedy Green]

Now, I've already set the proxy so that it can connect to outside web
pages (I've already tried it out on more mundane pages with no
problems), so I'm not sure what else I need to do to be able to connect
to it to retrieve the data. Can anyone help me out? Many thanks!

Tackle a simpler problem first, e.g. read a perfectly vanilla web
page.

Then get a packet sniffer. See
http://mindprod.com/jgloss/ethereal.html

Then get your browser to poke at the weather site, and watch the
interactions. That will help you understand what you have to do.

If you are stuck, I could write you a class to handle this for a fee.