Preventing URLConnection from buffering entire output stream

[Mark Riordan]

In my client application, I upload huge files via HTTP.
The URLConnection class seems to automatically buffer
the entire contents, then when the OutputStream associated with
the URLConnection is closed, it computes the length
and prepends a Content-Length header before actually sending
any bytes to the web server.

This results in an OutOfMemoryException when the file is huge.

My application uses Transfer-Encoding: chunked, but
that doesn't help when the entire stream gets buffered anyway.

Is there a way around this without writing my own URLConnection
clone based on Sockets? Calling flush() on the OutputStream
seems to have no effect.

I understand that a workaround would be to use the new features
of Java 1.5, but that's too new to expect end users to have.

Any help would be appreciated.

Thanks!

Mark Riordan

Note: email address in header, (e-mail address removed), must have
"NoSpam" removed.

 

[Bruno Grieder]

Mark,

I believe, you can do this going at the lower [Client] Socket level.

Create a Client Socket.
Open the input and output streams.
Write the HTTP headers (a GET request I guess).
Read the Inputsream by chunks (you will need to remove the headers
though - anything to the blank line).
Close the socket.

Bruno

 

[Mark Riordan]

Bruno,

It is looking as if that's what I will have to do.
Too bad. I've done a lot of web searching, and I'm surprised
that this issue has not come up more frequently for Java programmers.

(In my case, it's a POST and the data is being sent
from my client rather than received, but it's the same idea.)

Thanks.

Mark

Mark,

I believe, you can do this going at the lower [Client] Socket level.

Create a Client Socket.
Open the input and output streams.
Write the HTTP headers (a GET request I guess).
Read the Inputsream by chunks (you will need to remove the headers
though - anything to the blank line).
Close the socket.

Bruno

In my client application, I upload huge files via HTTP.
The URLConnection class seems to automatically buffer
the entire contents, then when the OutputStream associated with
the URLConnection is closed, it computes the length
and prepends a Content-Length header before actually sending
any bytes to the web server.

This results in an OutOfMemoryException when the file is huge.

My application uses Transfer-Encoding: chunked, but
that doesn't help when the entire stream gets buffered anyway.

Is there a way around this without writing my own URLConnection
clone based on Sockets? Calling flush() on the OutputStream
seems to have no effect.

I understand that a workaround would be to use the new features
of Java 1.5, but that's too new to expect end users to have.

Any help would be appreciated.

Thanks!

Mark Riordan

Note: email address in header, (e-mail address removed), must have
"NoSpam" removed.

 

[Ann]

Bruno,

It is looking as if that's what I will have to do.
Too bad. I've done a lot of web searching, and I'm surprised
that this issue has not come up more frequently for Java programmers.

(In my case, it's a POST and the data is being sent
from my client rather than received, but it's the same idea.)

Thanks.

Mark

Mark,

I believe, you can do this going at the lower [Client] Socket level.

Create a Client Socket.
Open the input and output streams.
Write the HTTP headers (a GET request I guess).
Read the Inputsream by chunks (you will need to remove the headers
though - anything to the blank line).
Close the socket.

Bruno

In my client application, I upload huge files via HTTP.
The URLConnection class seems to automatically buffer
the entire contents, then when the OutputStream associated with
the URLConnection is closed, it computes the length
and prepends a Content-Length header before actually sending
any bytes to the web server.

This results in an OutOfMemoryException when the file is huge.

My application uses Transfer-Encoding: chunked, but
that doesn't help when the entire stream gets buffered anyway.

Is there a way around this without writing my own URLConnection
clone based on Sockets? Calling flush() on the OutputStream
seems to have no effect.

I understand that a workaround would be to use the new features
of Java 1.5, but that's too new to expect end users to have.

Any help would be appreciated.

Thanks!

Mark Riordan

Note: email address in header, (e-mail address removed), must have
"NoSpam" removed.

Excuse my naiveté, but why does a POST request need to
send the size?

 

[Mark Riordan]

Excuse my naiveté, but why does a POST request need to
send the size?

Technically, it doesn't.
Which is why I'm irritated that Sun's HttpURLConnect apparently
is hard-coded to always send Content-Length at the beginning.
(I say "apparently"; I've love for someone to prove me wrong.)

Now, the web server needs to have some way of figuring out
where the POST data end. But there are ways around this:
the headers "Transfer-Encoding: chunked", and "Connection: close".

However, neither of these techniques seems to affect HttpURLConnection's
unfortunate insistence on sending Content-Length anyway.

Mark R remove NoSpam from email addr

 

[Carl Howells]

Is there a way around this without writing my own URLConnection
clone based on Sockets? Calling flush() on the OutputStream
seems to have no effect.

I understand that a workaround would be to use the new features
of Java 1.5, but that's too new to expect end users to have.

Any help would be appreciated.

I haven't investigated in this context, but I've used the jakarta
project's HTTPClient Library before, and found it much more flexible
than HttpURLConnection. Have you taken a look at it?

 

[Mark Riordan]

Thanks for the tip. I found some other HTTP libraries
that looked good, but their licensing terms were not as
generous as that of Jakarta.

Mark R