send and receive a large byte[] over network?

[knguyen]

Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks

 

[Joshua Cranmer]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

It is hard to suggest the ideal solution when the problem is this
underspecified, but I will stick with the most generic solution:

Open up a Socket to the desired computer and port, take its inputStream,
and then use the write(byte[]) method of said stream to send the data.

If the bandwidth is insufficient, Java has built-in GZIP compression.

 

[Daniel Pitts]

Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks

300k isn't that large. I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up into
smaller chunks, OutputStream allows you to do that.

 

[Arne Vajhøj]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

300 KB is not large.

What have you tried ?

It is relative straightforward.

Below is attached a very simple example.

Arne

=================================

import java.io.*;
import java.net.*;

public class Server {
public static void main(String[] args) throws Exception {
ServerSocket ss = new ServerSocket(9999);
Socket s = ss.accept();
InputStream is = s.getInputStream();
OutputStream os = new FileOutputStream("C:\\z2.zip");
byte[] b = new byte[10000];
int n;
while((n = is.read(b)) >= 0) {
os.write(b, 0, n);
}
os.close();
is.close();
}
}

import java.io.*;
import java.net.*;

public class Client {
public static void main(String[] args) throws Exception {
Socket s = new Socket("localhost", 9999);
OutputStream os = s.getOutputStream();
InputStream is = new FileInputStream("C:\\z1.zip");
byte[] b = new byte[10000];
int n;
while((n = is.read(b)) >= 0) {
os.write(b, 0, n);
}
os.close();
is.close();
}
}

 

[Mark Space]

Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks

All of the above are good. A basic socket should work fine.

Have you considered how you will test this? A basic part of software
design is proper testing. Testing works best if you design the tests
before you start.

Consider: 100T = 100 b/s = 12,500 bytes per second
300,000 / 12,500 = 24 seconds.

1,000T (giga bit) should be about 2.4 seconds.

You should expect lower numbers (as much as 3 times longer if your
network topology is lacking) but not more than about 3x. So test to
make sure your number hit this range. If they don't, try to find out why.

If you're using 10T, it might be faster to burn a CD. ;-)

 

[Lew]

Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks

300k isn't that large. I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up into
smaller chunks, OutputStream allows you to do that.

That's 300k elements - what is the size of an element?

A 1-kB element would mean a 300 MB transfer. That's an hour's transfer.

Maybe torrent?

If you aren't in a huge hurry and just want to keep your TCP/IP pipe free,
snail-mail a CD. (If it's a LAN, sneaker-net the CD.)

 

[Arne Vajhøj]

Consider: 100T = 100 b/s = 12,500 bytes per second
300,000 / 12,500 = 24 seconds.

1,000T (giga bit) should be about 2.4 seconds.

You should expect lower numbers (as much as 3 times longer if your
network topology is lacking) but not more than about 3x. So test to
make sure your number hit this range. If they don't, try to find out why.

If you're using 10T, it might be faster to burn a CD. ;-)

????

10/100/1000 is millions bits per second not bits per second.

Arne

 

[Arne Vajhøj]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

300k isn't that large. I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up
into smaller chunks, OutputStream allows you to do that.

That's 300k elements - what is the size of an element?

Unless somebody tell us otherwise we are assuming that the
elements are in the array mentioned ...

Arne

 

[Daniel Pitts]

Hi,

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Thanks

300k isn't that large. I would suggest using an OutputStream, and
simply writing the array to that stream. If you need to break it up
into smaller chunks, OutputStream allows you to do that.

That's 300k elements - what is the size of an element?

A 1-kB element would mean a 300 MB transfer. That's an hour's transfer.

Maybe torrent?

If you aren't in a huge hurry and just want to keep your TCP/IP pipe
free, snail-mail a CD. (If it's a LAN, sneaker-net the CD.)

byte[] = new byte[300000]; // 300k elements.
Even at 5k/s, thats only one minute.

Even a 300MB transfer on modern connections is around 5 minutes.

In any case, opening a socket, and writing to it is probably the best
approach. If you need the ability to resume after disconnect, you might
look into using FTP or some similar protocol. Or, create your own
simple protocol.

 

[Lew]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am

That's 300k elements - what is the size of an element?

Unless somebody tell us otherwise we are assuming that the
elements are in the array mentioned ...

Oh, yeah. Duhy. Thanks. That's a millihour's transfer, ...

 

[George Neuner]

Consider: 100T = 100 b/s = 12,500 bytes per second
300,000 / 12,500 = 24 seconds.

1,000T (giga bit) should be about 2.4 seconds.

You should expect lower numbers (as much as 3 times longer if your
network topology is lacking) but not more than about 3x. So test to
make sure your number hit this range. If they don't, try to find out why.

If you're using 10T, it might be faster to burn a CD. ;-)

10T = 10Mb/s = 1MB/s.
100T = 100Mb/s = 10MB/s.
1000T = 1000Mb/s = 100MB/s.

Including protocol overhead on a typical 10T Ethernet, 300KB should
take less than 1/2 second.

George

 

[Mark Space]

10T = 10Mb/s = 1MB/s.
100T = 100Mb/s = 10MB/s.
1000T = 1000Mb/s = 100MB/s.

Oops, yeah I think I had a brain fart there....

 

[Roedy Green]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Use a socket and a DataOutputStream. See
http://mindprod.com/applet/fileio.html for sample code.

 

[Arne Vajhøj]

Would anyone please suggest me a solution to send/receive a large
byte[] (about 300.000 elements) over network? Currently, I am
clueless.

Use a socket and a DataOutputStream.

Why ?

The service DataOutputStream provide over any OutputStream
is the ability to send items bigger than byte and handle
endianess.

Arne

 

[Roedy Green]

Why ?

The service DataOutputStream provide over any OutputStream
is the ability to send items bigger than byte and handle
endianess.

The code is compact. It does not require any parsing. It has low
overhead to prepare the stream.

I did not suggest an ObjectStream because you might overflow it with
such a long stream. It depends on the structure.

XML would be obscenely large in comparison and again might overflow
in-ram tree representations.

About your only problem with the DataOutputStream is making sure the
code on both ends is the exact same version. Unfortunately you can use
a common list of fields for both read and write the way you can in
languages such as Abundance.

 

[Roger Lindsjö]

The code is compact. It does not require any parsing. It has low
overhead to prepare the stream.

I did not suggest an ObjectStream because you might overflow it with
such a long stream. It depends on the structure.

XML would be obscenely large in comparison and again might overflow
in-ram tree representations.

About your only problem with the DataOutputStream is making sure the
code on both ends is the exact same version. Unfortunately you can use
a common list of fields for both read and write the way you can in
languages such as Abundance.

I read this as Arne Vajhøj meaning that the poster could just use the
regular write(byte[]) on the output stream given by the socket. Or are
you saying that the write(byte[]) has limits in the data sent (apart
from the maximum size of a byte array)?

//Roger Lindsjö

 

[Arne Vajhøj]

The code is compact. It does not require any parsing. It has low
overhead to prepare the stream.

It does not provide anything useful for the problem at hand
and is pure code clutter.

The OutputStream returned from the Socket has a write method that
is perfect for the task.

Arne