java download manager

[Jean-Daniel Gamache]

Hi,

I want to write a Java application that can download a file (url)
fast.

To do this, I want to open multiple connections on the same file(url)
and each connection will download a chunk of the file.

Any clue how I can do this ?

thanks,
JD

 

[Roedy Green]

To do this, I want to open multiple connections on the same file(url)
and each connection will download a chunk of the file.

Any clue how I can do this ?

see http://mindprod.com/fileio.html

Tell it you want to do a CGI get.

Look up the HTTP headers to control the offset and length of the chunk
you want, and spin off three threads after you figure out how big the
whole thing is with a HEAD.

 

[Alan Meyer]

Hi,

I want to write a Java application that can download a file (url)
fast.

To do this, I want to open multiple connections on the same file(url)
and each connection will download a chunk of the file.

Any clue how I can do this ?

thanks,
JD

Note: I would expect your technique to be SLOWER,
not faster.

The speed of the download will almost certainly be
determined by the speed of the physical link between
the two computers. Creating two separate TCP/IP
connections over the same link (e.g., modem and
telephone line, modem and cable, etc.) will increase
the overhead, i.e., the number of bits dedicated to
connection creation and control, but will not increase
the total number of bits per second that can flow over
the link. So a smaller number of data bits from the file
you want will be downloaded per second than if you
had one connection with more of the bits dedicated
to data instead of control.

If you have two physical links, e.g., two phone lines
that you can use at once, that's another story.

However, even in that case, you'd also need to get
the server to cooperate with you. To the best of
my knowledge, neither the http nor the ftp protocols
will download two halves of a file in this way. So
you'd need a program on the server to split the
file in half into two separate files - then go after
each half on each connection. If it's a server that's
not under your control, I don't think you'll be able
to do that.

Alan

 

[Carl Howells]

Note: I would expect your technique to be SLOWER,
not faster.

[snip]


However, even in that case, you'd also need to get
the server to cooperate with you. To the best of
my knowledge, neither the http nor the ftp protocols
will download two halves of a file in this way. So
you'd need a program on the server to split the
file in half into two separate files - then go after
each half on each connection. If it's a server that's
not under your control, I don't think you'll be able
to do that.

Well, that technique has made a significant speed increase in the past,
due to a couple of factors you're missing.

First, many servers cap their upload speed *per connection*. If you
open multiple connections, you can get much greater speed, up to the
available bandwidth between your computer and the server.

Second, the server does cooperate with the user on this, if it supports
"resume" style functionality. Many web and FTP servers do support
resuming a download, and therefore support this technique.

And third... Many programs made when this style of download manager was
popular included functionality to search for the same file on multiple
servers, and download different parts of it from different servers.
When this worked correctly (not that often, which is why it's pretty
much abandoned for this type of downloading now), it also provided a
giant speed boost.