Most efficient way to transfer a file directory recursively (using sockets)

[Arash Nikkar]

I am trying to build a simple application which can be used much like a
ftp application, where the user will drag files over, and it will be
uploaded to a server. So I have spent the last day trying to figure out
the best/most efficient way to do this, and I have in a sense come up
with the following 3 choices

1. Create a client/server connection, transfer files using a
DataOutputStream, which means each file transfer will need to be split
into 2 main parts, the first is the pass the filename, the second is to
actually send the file (obviously this will be split into smaller
pieces as well)

2. This option is the same as above, but it would use NIO with
FileChannel and SocketChannel (something I haven't quite figured out
yet)

3. The last way would be to use an ObjectOutputStream, and keep the
File object information intact, which would make writing the file to
the server, and maintaining directory hierarchy much easier. However,
I'm pretty sure this would yield the slowest transfer rates.

I was wondering if anyone had any suggestions? Better yet, if someone
had an example of a client/server transfer using NIO & FileChannel.

your guidance is appreciated

 

[Red Orchid]

I am trying to build a simple application which can be used much like a
ftp application, where the user will drag files over, and it will be
uploaded to a server.

To my thinking ....

<code>

//
// Main routine
//
OutputStream ou = ... // from socket.

File[] fs = getAllSendingFiles();

constructHead(ou, fs);
constructBody(ou, fs);
ou.flush();

// close stream and socket
....


//
// functions
//
File[] getAllSendingFiles() {
// Maybe recursive is needed.
return ...;
}

void constructHead(OutputStream out, File[] fs) throws IOException {

StringBuilder head = new StringBuilder(8196);

head.append("Protocol: Yours 1.0");
head.append(CRLF);

head.append("-"); // divider 1
head.append(CRLF);

for (File f : fs) {

addFileHead(head, f);
}
head.append("---"); // divider 3
head.append(CRLF);

out.write(head.toString().getBytes("utf-8"));
}

void addFileHead(StringBuilder head, File f) {

head.append("Path: " + f.getPath());
head.append(CRLF);

head.append("Length: " + f.length());
head.append(CRLF);

// head.append("Type: Binary"); // or Text/UTF-8
// head.append(CRLF);

head.append("--"); // divider 2
head.append(CRLF);
}

void constructBody(OutputStream ou, File[] fs) throws IOException {

for (File f : fs) {

ou.write(getFileBytes(f));
}
}

byte[] getFileBytes(File f) throws IOException {
// from FileInputStream..
return ...;
}

</code>

 

[Arash Nikkar]

Hi Red,

Thanks for the reply. One question though.

in the construct head method, what is all the CRLF stuff? thanks!

I am trying to build a simple application which can be used much like a
ftp application, where the user will drag files over, and it will be
uploaded to a server.

To my thinking ....

<code>

//
// Main routine
//
OutputStream ou = ... // from socket.

File[] fs = getAllSendingFiles();

constructHead(ou, fs);
constructBody(ou, fs);
ou.flush();

// close stream and socket
...


//
// functions
//
File[] getAllSendingFiles() {
// Maybe recursive is needed.
return ...;
}

void constructHead(OutputStream out, File[] fs) throws IOException {

StringBuilder head = new StringBuilder(8196);

head.append("Protocol: Yours 1.0");
head.append(CRLF);

head.append("-"); // divider 1
head.append(CRLF);

for (File f : fs) {

addFileHead(head, f);
}
head.append("---"); // divider 3
head.append(CRLF);

out.write(head.toString().getBytes("utf-8"));
}

void addFileHead(StringBuilder head, File f) {

head.append("Path: " + f.getPath());
head.append(CRLF);

head.append("Length: " + f.length());
head.append(CRLF);

// head.append("Type: Binary"); // or Text/UTF-8
// head.append(CRLF);

head.append("--"); // divider 2
head.append(CRLF);
}

void constructBody(OutputStream ou, File[] fs) throws IOException {

for (File f : fs) {

ou.write(getFileBytes(f));
}
}

byte[] getFileBytes(File f) throws IOException {
// from FileInputStream..
return ...;
}

</code>

 

[Red Orchid]

in the construct head method, what is all the CRLF stuff?

CRLF is "\r\n".

 

[Oliver Wong]

I am trying to build a simple application which can be used much like a
ftp application, where the user will drag files over, and it will be
uploaded to a server. So I have spent the last day trying to figure out
the best/most efficient way to do this, and I have in a sense come up
with the following 3 choices

It's a tricky topic, so you're probably best using an existing protocol
(e.g. FTP) than reinventing the wheel. That said, in most cases, the CPU is
much faster than the network connection, so you might consider compressing
the file before sending it over.

For a more detailed answer, you might want to more formally define
"best/most efficient". Are you talking lowest latency? Lowest bandwidth
consumption? Lowest CPU consumption? Something else?

- Oliver

 

[Arash Nikkar]

Hi Oliver,

I know there are existing FTP protocols out there which I could
utilize, but since this is a personal project, I figured I should try
it out on my own for a bit.

by most efficient I was thinking fastest throughput (i.e. from start to
finish)...so it seems that zipping up the files might be the best
solution. I would also like to utilize NIO, as that seems to increase
throughput significantly.

 

[Arash Nikkar]

just wanted to *bump* this message after the long weekend

 

[Oliver Wong]

just wanted to *bump* this message after the long weekend

Is there an unaddressed question that you'd like answered?

- Oliver

 

[Arash Nikkar]

Hi Oliver,

sorry about that...confused this post with a different one...still
trying to figure out how to use FileChannels and SocketChannels, but I
should play with them a bit more before asking a question.