Using HTTP to return unspecified size of data in response..

[SPG]

Hi,

We have a requirement (mad as it may seem) where we need to do an HTTP POST
to subscribe to a datafeed.
The response from this feed then pushes data from a price feed.

Sounds simple doesn't it?
Well, it appears that the java.net.HttpUrlConnection will read the
"content-length" number of bytes from the response only.
Currently, our hack test-bed returns an HTTP 1.0 response which sets
content-length to 0.
Alternatively we have tried setting this length to a *massive* number. The
trouble then is with proxies.
We have found that proxiesdo horrid things to HTTP response like buffer it
for a nominal period, or until the response stream is closed. Now that is no
good to anyone.

So, my question is, how can I in effect stream an unspecified amount of
text/plain type data through an HTTP response?
I have tried looking at some of the HTTP specs flying around on GOOGLE, and
found a few useful hints but nothing concrete.

Here was a good start:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
I did find that using the "chunked" value in the "transfer-encoding" header,
but again, the HttpUrlConnection seems to be doing some strange things with
this.

Does anyone out there know of a fairly robust way of achieving this?

Steve

 

[Rogan Dawes]

Hi,

We have a requirement (mad as it may seem) where we need to do an HTTP POST
to subscribe to a datafeed.
The response from this feed then pushes data from a price feed.

Sounds simple doesn't it?
Well, it appears that the java.net.HttpUrlConnection will read the
"content-length" number of bytes from the response only.
Currently, our hack test-bed returns an HTTP 1.0 response which sets
content-length to 0.
Alternatively we have tried setting this length to a *massive* number. The
trouble then is with proxies.
We have found that proxiesdo horrid things to HTTP response like buffer it
for a nominal period, or until the response stream is closed. Now that is no
good to anyone.

So, my question is, how can I in effect stream an unspecified amount of
text/plain type data through an HTTP response?
I have tried looking at some of the HTTP specs flying around on GOOGLE, and
found a few useful hints but nothing concrete.

Here was a good start:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
I did find that using the "chunked" value in the "transfer-encoding" header,
but again, the HttpUrlConnection seems to be doing some strange things with
this.

Does anyone out there know of a fairly robust way of achieving this?

Steve

Chunked data is the right way of doing this. It is only supported by
HTTP/1.1 compliant browsers, though, which means that your program needs
to tell the server that it IS a 1.1 compliant browser somehow. AND,
obviously, your server needs to send Chunked data, rather than providing
a fixed length header field.

Then, you will probably need to implement your own chunked input stream
class that you would wrap around the URLConnections inputstream.

Alternatively, have a look at the various implementations of
ChunkedInputStream floating arond the net.

Rogan