2 question poll: large and long http response (stream like)

[NOBODY]

Hi,

We are arguing about the elegance/irregularity of the possibility to send
*gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

Assuming the client can handle it and that the tcp/ssl layer are reliable
and won't disconnect, would you say it is:

question 1
a-against the http spec (what section______)
b-supported/covered by http spec (chunked encoding)
c-other________

question 2:
a-risky (reason:_______)
b-unusual (reason:_______)
c-safe (reason:_______)
d-common (example:_____)
e-other_________


Thanks to all in advance.

 

[Rogan Dawes]

Hi,

We are arguing about the elegance/irregularity of the possibility to send
*gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

Assuming the client can handle it and that the tcp/ssl layer are reliable
and won't disconnect, would you say it is:

question 1
a-against the http spec (what section______)
b-supported/covered by http spec (chunked encoding)

I'd say it would be supported using chunked encoding.

c-other________

question 2:
a-risky (reason:_______)

What happens if there IS a disconnect for some reason? Can you resume
operations where you left off?

b-unusual (reason:_______)

Not unusual, per se. Internet routers, for example, create long-lived
TCP connections to exchange routing information. The volume may be
greater, but that's no matter.

c-safe (reason:_______)

Dunno about safe. See question about disconnects.

 

[John C. Bollinger]

We are arguing about the elegance/irregularity of the possibility to
send *gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

Assuming the client can handle it and that the tcp/ssl layer are
reliable and won't disconnect, would you say it is:

question 1
a-against the http spec (what section______)
b-supported/covered by http spec (chunked encoding)

I'd say it would be supported using chunked encoding.

I agree. The spec does not place any limit that I know of on the total
size of a message delivered via the chunked transfer coding. It says
little or nothing about the permissible time to deliver a complete response.

What happens if there IS a disconnect for some reason? Can you resume
operations where you left off?

Spot on. A service interruption probably means that you're hosed,
because as far as I can tell, there is no way to resume an HTTP transfer
in the middle. You could build a resumable protocol on top of HTTP, but
then you wouldn't be delivering the whole logical response in a single
HTTP response.

Not unusual, per se. Internet routers, for example, create long-lived
TCP connections to exchange routing information. The volume may be
greater, but that's no matter.

I disagree. Routers may establish long-lived connections amongst
themselves, but that's not the same thing as a continuous data stream
over a comparable period. At least it doesn't seem that way to me. The
proposed behavior is in any case assuredly unusual in the HTTP space.

 

[Rogan Dawes]

We are arguing about the elegance/irregularity of the possibility to
send *gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

What happens if there IS a disconnect for some reason? Can you resume
operations where you left off?

Spot on. A service interruption probably means that you're hosed,
because as far as I can tell, there is no way to resume an HTTP transfer
in the middle. You could build a resumable protocol on top of HTTP, but
then you wouldn't be delivering the whole logical response in a single
HTTP response.

Well, HTTP DOES have support for "partial content", for example, see the
"206 Partial Content" response status, and the "request ranges" support
defined in RFC2616 (I forget the exact header ;-)

I disagree. Routers may establish long-lived connections amongst
themselves, but that's not the same thing as a continuous data stream
over a comparable period. At least it doesn't seem that way to me. The
proposed behavior is in any case assuredly unusual in the HTTP space.

Essentially, I think that the OP is wanting to do something that
requires lots of data to be transferred, and wants to do it without
getting the firewall administrators involved. Hence the desire to do
this on top of HTTP . . .

Not necessarily a good idea, IMO.

Rogan

 

[Chris Uppal]

We are arguing about the elegance/irregularity of the possibility to send
*gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

I would worry about this. Specifically, I'd worry about 32-bit counters.

At the TCP level there /shouldn't/ be a problem. TCP does have a 32-bit
sequence number in it, but that is defined to wrap around. What /might/ be a
worry is any other code that is in the transmission path, anti-virus software,
anti-intrusion software, statefull firewalls, etc. The point is that although
these things /should/ handle that much data, there's always a chance that
they'll contain 32-bit counters that will cause problems if they wrap. I,
personally, wouldn't worry /too/ much about this since I would expect the
makers/writers of such tools to test long flows.

I have no idea how a multi-gigabyte flow would interact with the SSL protocol.
Nor what the security/crypto implications are (would you be effectively
supplying an attacker with a very large body of data all encrypted with the
same key ? I believe SSL has provision for negotiating a new key during a
session, but I don't know when/how/whether that happens).

At the HTTP level, there /shouldn't/ be a problem either (assuming chunked
transmission). This time, though, I wouldn't feel nearly as confident that
there would be no problems with 32-bit wraparound. Neither HTTP clients nor
HTTP servers are designed /specifically/ for that kind of load, and I wouldn't
be surprised if such conditions had received little or no testing (or, perhaps,
even thought). The same considerations apply to any other s/w that is working
at the HTTP level and which is involved in the execution and/or transmission
path.

-- chris

 

[Chris Uppal]

Spot on. A service interruption probably means that you're hosed,
because as far as I can tell, there is no way to resume an HTTP transfer
in the middle.

I thought there was some sort of request-range: header in HTTP 1.1 ? I admit
that I'm too lazy to go look it up, so that may be complete bollocks...

But even if it did exist, the question would arise: will the actual
implementations work with ranges that can't be expressed in 32-bits ?

-- chris

 

[NOBODY]

Essentially, I think that the OP is wanting to do something that

requires lots of data to be transferred, and wants to do it without
getting the firewall administrators involved. Hence the desire to do
this on top of HTTP . . .

Not necessarily a good idea, IMO.

Rogan

Hehehe...
As a coder, I would go straight to TCP.
But yeah... I'm JACWABB (just a coder with a boss behind...)

It's more like streaming data to simple clients, like applets, perl, wget
to console, etc... Any easy-to-code data consumer.

You're thinking: "but why don't you make many shorter connections?"
Well, for now, we would really like not to 'skip a beat' because the
streamed data in question is a flow of events. Having a 'mark' in the
protocol (to resume at requested position) would suddenly mean extra
bufferring and connection subscription/management code and ...
aaaahhhhAAHH...(sigh...) while a non-expected loss of connection is not
going to require lossless recovery. We would just tell the user, and he
would restart.

 

[NOBODY]

We are arguing about the elegance/irregularity of the possibility to
send *gigs and gigs of data
*during many days
*over a single https response.
*[with tomcat5 & java 1.4.2]

I would worry about this. Specifically, I'd worry about 32-bit
counters.

At the TCP level there /shouldn't/ be a problem. TCP does have a
32-bit sequence number in it, but that is defined to wrap around.
What /might/ be a worry is any other code that is in the transmission
path, anti-virus software, anti-intrusion software, statefull
firewalls, etc. The point is that although these things /should/
handle that much data, there's always a chance that they'll contain
32-bit counters that will cause problems if they wrap. I, personally,
wouldn't worry /too/ much about this since I would expect the
makers/writers of such tools to test long flows.

I have no idea how a multi-gigabyte flow would interact with the SSL
protocol. Nor what the security/crypto implications are (would you be
effectively supplying an attacker with a very large body of data all
encrypted with the same key ? I believe SSL has provision for
negotiating a new key during a session, but I don't know
when/how/whether that happens).

At the HTTP level, there /shouldn't/ be a problem either (assuming
chunked transmission). This time, though, I wouldn't feel nearly as
confident that there would be no problems with 32-bit wraparound.
Neither HTTP clients nor HTTP servers are designed /specifically/ for
that kind of load, and I wouldn't be surprised if such conditions had
received little or no testing (or, perhaps, even thought). The same
considerations apply to any other s/w that is working at the HTTP
level and which is involved in the execution and/or transmission path.

-- chris

Thanks for the details.

I keep my eye on tomcat: will the httpsession stay alive during that
time, and if not, will the socket be forcibly closed... I think not, but
eh, dunno. For sure, the CLF web log will look funny...!
ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...!

 

[John C. Bollinger]

John C. Bollinger wrote:




I thought there was some sort of request-range: header in HTTP 1.1 ? I admit
that I'm too lazy to go look it up, so that may be complete bollocks...

Ah, um, yes; good point. Time to go back and reread the spec again.
Request ranges might not really help, however, if the response is
dynamically generated. That is especially true if it is a real-time
stream. (E.g. data from a scientific instrument.)

 

[Chris Uppal]

. For sure, the CLF web log will look funny...!
ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...!

And if it wraps to negative numbers, then your logfile analyser will have
kittens ;-)

-- chris

 

[John C. Bollinger]

I keep my eye on tomcat: will the httpsession stay alive during that
time, and if not, will the socket be forcibly closed... I think not, but
eh, dunno. For sure, the CLF web log will look funny...!
ex: 200 ok, 293'465'023 milliseconds, 19'535'487'623 bytes...!

This is a Tomcat implementation detail, and I don't have the familiarity
with Tomcat's source that would be necessary to answer it. I find it
highly unlikely that Tomcat would forcibly close the connection in the
middle of a response, but it is possible that the user would end up in a
new session if they had to reconnect after even a momentary
interruption. This should be fairly easy to test.

 

[Rogan Dawes]

Hehehe...
As a coder, I would go straight to TCP.
But yeah... I'm JACWABB (just a coder with a boss behind...)

It's more like streaming data to simple clients, like applets, perl, wget
to console, etc... Any easy-to-code data consumer.

You're thinking: "but why don't you make many shorter connections?"
Well, for now, we would really like not to 'skip a beat' because the
streamed data in question is a flow of events. Having a 'mark' in the
protocol (to resume at requested position) would suddenly mean extra
bufferring and connection subscription/management code and ...
aaaahhhhAAHH...(sigh...) while a non-expected loss of connection is not
going to require lossless recovery. We would just tell the user, and he
would restart.

One last thing to consider:

If you expect to have some idle time during the transfer, and you will
be going through proxies, you may also have to deal with the proxy
thinking that the connection has gone idle, and disconnecting you.

Many proxies will time out a persistent connection if there is no data
trasferred for more than about 30 seconds . . .

Rogan