network effeciently

Discussion in 'Java' started by Parker Thompson, May 11, 2004.

  1. Hello,

    I am trying to send files over the network using tcp sockets and am
    getting horrible speeds even over fast connections, perhaps 10k/sec where
    I would expect at least a factor of 10 greater. I feel like this should
    be pretty simple, but can't seem to figure it out.

    My code is as follows:

    //***Receiver code***
    InputStream netIn = connectionSocket.getInputStream();
    BufferedInputStream bIn = new BufferedInputStream(netIn);

    FileOutputStream fileOut = new FileOutputStream(outputFile);
    BufferedOutputStream bufOut = new BufferedOutputStream(fileOut);

    int b;
    while((b = bIn.read()) >= 0){
    bufOut.write(b);
    receivedBytes++;

    if(abort){
    cleanup_upon_failure();
    }
    }
    bufOut.flush();
    bufOut.close();
    netIn.close();
    fileOut.close();


    //***Sender code***
    OutputStream out = connection.getOutputStream();
    BufferedOutputStream bOut = new BufferedOutputStream(out);

    FileInputStream fIn = new FileInputStream(fileToSend);
    BufferedInputStream bIn = new BufferedInputStream(fIn);

    while( bIn.available() > 0 ){
    bOut.write(bIn.read());
    }

    bOut.flush();
    out.close();

    Any suggestions are apreciated. Thanks,

    pt.
    Parker Thompson, May 11, 2004
    #1
    1. Advertising

  2. On Mon, 10 May 2004 22:56:30 -0700, Parker Thompson wrote:
    > I am trying to send files over the network using tcp sockets and am
    > getting horrible speeds even over fast connections, perhaps 10k/sec
    > where I would expect at least a factor of 10 greater. I feel like
    > this should be pretty simple, but can't seem to figure it out.
    >
    > My code is as follows:


    [...]

    > Any suggestions are apreciated. Thanks,


    Two quick observations:

    - read and write much more than one byte at a time. Look at methods
    BufferedInputStream.read(byte[], int off, int len) and
    BufferedOutputStream.write(byte[], int off, int len). Pay attention
    to the return value, which reports how many bytes were actually read
    or written.

    - don't use BufferedInputStream.available() before calling read().
    Just block in the call to read(), and you will receive the data as
    soon as it becomes available.

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, May 11, 2004
    #2
    1. Advertising

  3. Parker Thompson wrote:

    > I am trying to send files over the network using tcp sockets and am
    > getting horrible speeds even over fast connections, perhaps 10k/sec where
    > I would expect at least a factor of 10 greater. I feel like this should
    > be pretty simple, but can't seem to figure it out.


    Because you read & write 1 byte at time.
    Try to read many (e.g. 8K) bytes at time
    and you will see a major improvement.

    - Dario
    =?UTF-8?B?IkRhcmlvIChkcmlua2luZyBjb++sgGVlIGluIHRo, May 11, 2004
    #3
  4. Parker Thompson

    Rogan Dawes Guest

    Parker Thompson wrote:

    > Hello,
    >
    > I am trying to send files over the network using tcp sockets and am
    > getting horrible speeds even over fast connections, perhaps 10k/sec where
    > I would expect at least a factor of 10 greater. I feel like this should
    > be pretty simple, but can't seem to figure it out.
    >
    > My code is as follows:
    >
    > //***Receiver code***
    > InputStream netIn = connectionSocket.getInputStream();
    > BufferedInputStream bIn = new BufferedInputStream(netIn);
    >
    > FileOutputStream fileOut = new FileOutputStream(outputFile);
    > BufferedOutputStream bufOut = new BufferedOutputStream(fileOut);
    >
    > int b;
    > while((b = bIn.read()) >= 0){
    > bufOut.write(b);
    > receivedBytes++;
    >
    > if(abort){
    > cleanup_upon_failure();
    > }
    > }
    > bufOut.flush();
    > bufOut.close();
    > netIn.close();
    > fileOut.close();
    >
    >
    > //***Sender code***
    > OutputStream out = connection.getOutputStream();
    > BufferedOutputStream bOut = new BufferedOutputStream(out);
    >
    > FileInputStream fIn = new FileInputStream(fileToSend);
    > BufferedInputStream bIn = new BufferedInputStream(fIn);
    >
    > while( bIn.available() > 0 ){
    > bOut.write(bIn.read());
    > }
    >
    > bOut.flush();
    > out.close();
    >
    > Any suggestions are apreciated. Thanks,
    >
    > pt.


    Don't write your data one byte at a time. You are wasting a lot of time
    looping (and probably sending 1-byte packets with 32-48 bytes of
    overhead - I'm not sure of the size of a TCP header offhand).

    Rather write it in chunks, using the OutputStream.write(byte[]) method,
    or OutputStream.write(byte[], int, int) method.

    similarly, use the "int InputStream.read(byte[])" method to read from
    the stream.

    The MTU (Maximum Transmission Unit?) of an Ethernet LAN is typically
    1514 bytes, but the underlying TCP libraries will deal with that for
    you. Generally write in chunks of (the network MTU)-(the TCP header
    size) bytes to get the best speed.

    If you really need to get the best speed possible, over a variety of
    networks, you will need to do Path MTU Discovery to work out the biggest
    packet that can be sent over the entire path between client and server,
    and send chunks that size minus the size of the TCP header. I'm not sure
    if Java allows you to determine the MTU, though. Maybe via NIO?

    Regards,

    Rogan
    --
    Rogan Dawes

    *ALL* messages to will be dropped, and added
    to my blacklist. Please respond to "nntp AT dawes DOT za DOT net"
    Rogan Dawes, May 11, 2004
    #4
  5. Parker Thompson

    Chris Uppal Guest

    Gordon Beaton wrote:

    > - read and write much more than one byte at a time.


    It'd be interesting to know how much difference this really makes.

    The /point/ of the buffered streams is that there should be little difference
    between reading/writing one byte at a time vs. in large chunks. Note that word
    "should" ;-)

    I suppose I could go measure it, but I have other fish to fry today...[*]

    -- chris

    [*] but a quick glance at the Sun 1.4.2 source shows nothing too obvious -- the
    overhead of a synchronised method call per byte, and a test-and-branch.
    Chris Uppal, May 11, 2004
    #5
  6. Parker Thompson

    Yu Song Guest

    Parker Thompson wrote:
    > Hello,
    >
    > I am trying to send files over the network using tcp sockets and am
    > getting horrible speeds even over fast connections, perhaps 10k/sec where
    > I would expect at least a factor of 10 greater. I feel like this should
    > be pretty simple, but can't seem to figure it out.
    >
    > My code is as follows:
    >
    > //***Receiver code***
    > InputStream netIn = connectionSocket.getInputStream();
    > BufferedInputStream bIn = new BufferedInputStream(netIn);
    >
    > FileOutputStream fileOut = new FileOutputStream(outputFile);
    > BufferedOutputStream bufOut = new BufferedOutputStream(fileOut);
    >
    > int b;
    > while((b = bIn.read()) >= 0){
    > bufOut.write(b);
    > receivedBytes++;
    >
    > if(abort){
    > cleanup_upon_failure();
    > }
    > }
    > bufOut.flush();
    > bufOut.close();
    > netIn.close();
    > fileOut.close();
    >
    >
    > //***Sender code***
    > OutputStream out = connection.getOutputStream();
    > BufferedOutputStream bOut = new BufferedOutputStream(out);
    >
    > FileInputStream fIn = new FileInputStream(fileToSend);
    > BufferedInputStream bIn = new BufferedInputStream(fIn);
    >
    > while( bIn.available() > 0 ){
    > bOut.write(bIn.read());
    > }
    >
    > bOut.flush();
    > out.close();
    >
    > Any suggestions are apreciated. Thanks,
    >
    > pt.


    It could be 100 times faster(depends on your connection) if you
    read/write a chunk of data.


    --
    Song

    More info.:
    http://www.dcs.warwick.ac.uk/~esubbn/
    Yu Song, May 11, 2004
    #6
  7. On Tue, 11 May 2004 09:05:23 +0100, Chris Uppal wrote:
    > The /point/ of the buffered streams is that there should be little
    > difference between reading/writing one byte at a time vs. in large
    > chunks. Note that word "should" ;-)


    You could be right and that did occur to me too.

    I suppose it could be argued that the buffered streams are useful when
    you need to read small amounts of data at a time and don't want to be
    "penalized" for that.

    But that doesn't mean that you can be naive in the other cases.
    Regardless of networking issues, there is an additional cost of
    invoking the method once per byte instead of once per (e.g.) kilobyte.

    And, as usual when talking about optimizations, always measure! (No, I
    haven't).

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, May 11, 2004
    #7
  8. Parker Thompson

    Roedy Green Guest

    On Tue, 11 May 2004 09:22:29 +0100, Yu Song <> wrote or
    quoted :

    >It could be 100 times faster(depends on your connection) if you
    >read/write a chunk of data.


    There are two levels of overhead, talking to the socket, and talking
    to the buffered reader. Each kind of call has overhead. The bigger
    your buffer, the fewer calls to the underlying socket layer.

    If you are reading a giant chunk of raw bytes, DON'T use a
    BufferedInputStream. Use an unbuffered InputStream, and do it all in
    one giant I/O for maximum efficiency.

    see http://mindprod.com/jgloss/buffer.html for other hints on
    buffering.

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
    Roedy Green, May 11, 2004
    #8
  9. Parker Thompson

    Chris Uppal Guest

    Rogan Dawes wrote:

    > Don't write your data one byte at a time. You are wasting a lot of time
    > looping (and probably sending 1-byte packets with 32-48 bytes of
    > overhead - I'm not sure of the size of a TCP header offhand).


    This would be sensible and intelligent advice -- except that you haven't
    noticed that he's wrapping the raw networking streams in
    Buffered{Out/In}putStreams.

    See my other posts in this thread.

    -- chris
    Chris Uppal, May 11, 2004
    #9
  10. Parker Thompson

    Chris Uppal Guest

    Gordon Beaton wrote:

    > > The /point/ of the buffered streams is that there should be little
    > > difference between reading/writing one byte at a time vs. in large
    > > chunks. Note that word "should" ;-)

    >
    > I suppose it could be argued that the buffered streams are useful when
    > you need to read small amounts of data at a time and don't want to be
    > "penalized" for that.


    That's how I see it. Their job is to save /me/ from having to mess with
    buffering.


    > But that doesn't mean that you can be naive in the other cases.
    > Regardless of networking issues, there is an additional cost of
    > invoking the method once per byte instead of once per (e.g.) kilobyte.


    But in comparison with network speeds it seems unlikely to have much effect. I
    agree that being naive about it is not good though.


    > And, as usual when talking about optimizations, always measure! (No, I
    > haven't).


    Ah well, I did after all. On my machine (a 1.x Gz laptop -- I've forgotten
    what 'x' is), and running Java 1.5.0(beta1) I find that
    BufferedOutputStream.write(int) adds around 12 nanosecs of overhead per byte.
    (code appended to this message).

    That's four orders of magnitude too small to explain the effect he's seeing !

    I suspect that either his real code has a bug that isn't evident in the posted
    code (accidentally not using the buffered streams ?), or perhaps that the call
    to
    available() on a buffered stream wrapped around a FileStream is somehow
    immensely slow (!?).

    -- chris


    ======== code ===========
    import java.io.*;

    public class NullStream
    extends OutputStream
    {
    long m_written;

    public void
    write(int ch)
    {
    m_written++;
    }

    public void
    write(byte[] buffer, int offset, int length)
    {
    m_written += length;
    }


    static final long SHORT_LOOPS = 100000000L;
    static final long LONG_LOOPS = SHORT_LOOPS * 1000;


    public static void
    main(String[] args)
    throws IOException
    {
    long start = 0;
    long stop = 0;
    byte[] buffer = new byte[1024 * 64];
    byte b = 66;
    OutputStream ns = new NullStream();
    OutputStream bos = new BufferedOutputStream(ns);
    long i;

    start = System.currentTimeMillis();
    for (i = 0; i < SHORT_LOOPS; i++)
    ns.write(b);
    stop = System.currentTimeMillis();
    report(stop-start, "NullStream.write(int)", i);

    start = System.currentTimeMillis();
    for (i = 0; i < SHORT_LOOPS; i++)
    bos.write(b);
    stop = System.currentTimeMillis();
    report(stop-start, "BufferedOutputStream.write(int)", i);

    start = System.currentTimeMillis();
    for (i = 0; i < LONG_LOOPS; i += buffer.length)
    ns.write(buffer);
    stop = System.currentTimeMillis();
    report(stop-start, "NullStream.write(byte[])", i);

    start = System.currentTimeMillis();
    for (i = 0; i < LONG_LOOPS; i += buffer.length)
    bos.write(buffer);
    stop = System.currentTimeMillis();
    report(stop-start, "BufferedOutputStream.write(byte[])", i);
    }

    static void
    report(long msec, String test, long written)
    {
    double secs = msec / 1000.0;

    System.out.print(test);
    System.out.print(": ");
    System.out.print(secs);
    System.out.print(" seconds");
    if (secs > 0)
    {
    System.out.print(" = ");
    System.out.print(written / secs / 1000000);
    System.out.print(" MBytes/sec");
    }
    System.out.println();
    }
    }
    ======== code ===========
    Chris Uppal, May 11, 2004
    #10
  11. Parker Thompson

    Andy Fish Guest

    <snipped everything>
    It seems very unlikely that the problem is happening inside java - with
    throughputs as low as the OP is getting, you would have time to put each
    character in a DOM tree then process it with XSL before you started hitting
    CPU performance problems.

    The best way to investigate is to watch the CPU usage and monitor the total
    number of bytes on the network interface. if neither of these is maxing out
    I would suspect you have a serious synchronization problem (or sleep
    statement :) in your program

    Andy
    Andy Fish, May 11, 2004
    #11
  12. In our last episode Chris Uppal exclaimed:

    :> Don't write your data one byte at a time. You are wasting a lot of time
    :> looping (and probably sending 1-byte packets with 32-48 bytes of
    :> overhead - I'm not sure of the size of a TCP header offhand).
    :
    :This would be sensible and intelligent advice -- except that you haven't
    :noticed that he's wrapping the raw networking streams in
    :Buffered{Out/In}putStreams.

    A question on this point: does the buffer size have an impact? I'm
    unclear how the in/out stream buffers affect the size of the tcp packets.
    Should I be setting a non-default buffer size, or perhaps just skip
    buffering alltogether so I know what's going on?

    Thanks,

    pt.
    Parker Thompson, May 11, 2004
    #12
  13. Roedy Green wrote:

    > On Tue, 11 May 2004 09:22:29 +0100, Yu Song <> wrote or
    > quoted :
    >
    >
    >>It could be 100 times faster(depends on your connection) if you
    >>read/write a chunk of data.

    >
    >
    > There are two levels of overhead, talking to the socket, and talking
    > to the buffered reader. Each kind of call has overhead. The bigger
    > your buffer, the fewer calls to the underlying socket layer.
    >
    > If you are reading a giant chunk of raw bytes, DON'T use a
    > BufferedInputStream. Use an unbuffered InputStream, and do it all in
    > one giant I/O for maximum efficiency.
    >
    > see http://mindprod.com/jgloss/buffer.html for other hints on
    > buffering.


    Roedy:

    I didn't think that unbuffered would be as good as buffered so I wrote a
    couple of test programs and found that reading a whole array at a time
    was faster than single bytes but that buffering single byte reads was
    almost as good as buffered or multibyte. The difference between
    buffered and unbuffered multibyte reads was small but unbuffered was
    definitely faster.

    Here are my test programs.

    import java.io.*;
    import java.net.*;
    import java.util.*;

    public class TestServer {
    public static void main(String[] args) {
    byte[] b = new byte[1000];
    Random r = new Random();
    while (true) {
    try {
    ServerSocket ss = new ServerSocket(8192);
    System.out.println("Waiting for a connection");
    Socket s = ss.accept();
    System.out.println("Connected to: " +
    s.getInetAddress().getHostAddress());
    ss.close();
    OutputStream os = s.getOutputStream();
    BufferedOutputStream bos = new BufferedOutputStream(os);
    for (int i=0; i<5000; i++) {
    r.nextBytes(b);
    bos.write(b,0,b.length);
    }
    os.close();
    s.close();
    } catch (IOException ioe) {
    System.out.println(ioe);
    }
    System.out.println("Done\n");
    }
    }
    }

    import java.io.*;
    import java.net.*;

    public class TestClient {
    public static void main(String[] args) {
    try {
    byte[] buf = new byte[1000];
    int b;
    long start,stop;

    String host = (String)args[0];
    int port = Integer.parseInt(args[1]);
    Socket s = new Socket(host,port);
    System.out.println("Connected");
    InputStream is = s.getInputStream();
    start = System.currentTimeMillis();
    while ((b = is.read()) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("Unbuffered, one byte at a time: " +
    (stop - start) / 1000.0);
    is.close();
    s.close();
    System.out.println("Done\n");

    s = new Socket(host,port);
    System.out.println("Connected");
    is = s.getInputStream();
    BufferedInputStream bis = new BufferedInputStream(is);
    start = System.currentTimeMillis();
    while ((b = bis.read()) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("Buffered, one byte at a time: " +
    (stop - start) / 1000.0);
    bis.close();
    s.close();
    System.out.println("Done\n");

    s = new Socket(host,port);
    System.out.println("Connected");
    is = s.getInputStream();
    bis = new BufferedInputStream(is);
    start = System.currentTimeMillis();
    while ((b = bis.read(buf)) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("Buffered, 1000 bytes at a time: " +
    (stop - start) / 1000.0);
    bis.close();
    s.close();
    System.out.println("Done\n");

    s = new Socket(host,port);
    System.out.println("Connected");
    is = s.getInputStream();
    start = System.currentTimeMillis();
    while ((b = is.read(buf)) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("UnBuffered, 1000 bytes at a time: " +
    (stop - start) / 1000.0);
    is.close();
    s.close();
    System.out.println("Done\n");
    } catch (Exception e) {
    System.out.println(e);
    }
    }
    }



    --

    Knute Johnson
    email s/nospam/knute/
    Molon labe...
    Knute Johnson, May 12, 2004
    #13
  14. Parker Thompson

    Roedy Green Guest

    On Wed, 12 May 2004 01:10:50 GMT, Knute Johnson
    <> wrote or quoted :

    >I didn't think that unbuffered would be as good as buffered so I wrote a
    >couple of test programs and found that reading a whole array at a time
    >was faster than single bytes but that buffering single byte reads was
    >almost as good as buffered or multibyte. The difference between
    >buffered and unbuffered multibyte reads was small but unbuffered was
    >definitely faster.


    That sort of test you must run over and over and average the results.

    I'm surprised at your result. How many times did you run the test?

    --
    Canadian Mind Products, Roedy Green.
    Coaching, problem solving, economical contract programming.
    See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
    Roedy Green, May 12, 2004
    #14
  15. Roedy Green wrote:
    > On Wed, 12 May 2004 01:10:50 GMT, Knute Johnson
    > <> wrote or quoted :
    >
    >
    >>I didn't think that unbuffered would be as good as buffered so I wrote a
    >>couple of test programs and found that reading a whole array at a time
    >>was faster than single bytes but that buffering single byte reads was
    >>almost as good as buffered or multibyte. The difference between
    >>buffered and unbuffered multibyte reads was small but unbuffered was
    >>definitely faster.

    >
    >
    > That sort of test you must run over and over and average the results.
    >
    > I'm surprised at your result. How many times did you run the test?
    >


    I tried it a few times. It sends 5 megabytes of random bytes. I'll try
    it with 500mb and see how that turns out.

    --

    Knute Johnson
    email s/nospam/knute/
    Molon labe...
    Knute Johnson, May 12, 2004
    #15
  16. Parker Thompson

    Chris Uppal Guest

    Parker Thompson wrote:

    > > This would be sensible and intelligent advice -- except that you haven't
    > > noticed that he's wrapping the raw networking streams in
    > > Buffered{Out/In}putStreams.

    >
    > A question on this point: does the buffer size have an impact?


    Whatever effect it may have, it's unlikely to explain what you are seeing.

    I haven't checked the code, but I thing it would be quite difficult to persuade
    a BufferedOuputStream to send very small IP packets, at least if you are in a
    tight loop trying to write to the network. For instance, if you created the
    stream with a 1-byte buffer -- surely the worst possible case -- then went into
    a loop trying to send data through it, then
    - Each byte would be sent to the underlying
    Java network code individually.
    - The underlying code would /probably/ have no
    internal buffering and so would send each byte
    directly to the OS.
    - The OS would buffer up the data using "Nagle's
    algorithm" and so end up putting it on the wire in
    full sized IP packets anyway.

    ("Nagle's algorithm", is the process where the OS doesn't send data out
    immediately if that data would not fill an IP packet, but instead waits for a
    short time (a few millisecs I think) to see if the application will add any
    more data. It's controlled by one of the socket options, but is turned on by
    default).

    That would have considerable overhead from all the extra system calls, which
    would definitely slow things down, but not to the extent where it was taking
    around 100 microseconds to go round each iteration of your loop.


    > I'm
    > unclear how the in/out stream buffers affect the size of the tcp packets.


    Unless you disable Nagle's algorithm, it's very difficult to control the size
    of the packets. Unless you have an application that needs to control /when/
    the data is sent with a granularity of less than a few millisecs (which is rare
    but not unheard of -- I needed it once) then Nagle is your friend.


    > Should I be setting a non-default buffer size, or perhaps just skip
    > buffering alltogether so I know what's going on?


    These are both options that you may want to consider *after* you've found and
    fixed whatever is causing the problems you are seeing. The way you have the
    code now (judging by the extracts you posted) is fine as is. It is clear and
    simple and should be reasonably efficient. As with all optimisation, you do it
    *after* the code is working and in response to an /actual, measurable/
    performance problem. You don't go writing complicated code just for the hell
    of it (though it would seem that some of the other posters in this thread don't
    agree ;-) The measurements I posted earlier indicate that it highly unlikely
    that the use of single-byte writes into a buffered stream can cause the kind of
    problems you are seeing (remember *four orders of magnitude* !), so I think you
    have to look elsewhere. The advice Andy Fish posted about checking to see
    what's maxed-out is good.

    Still, if you want to see what's going on the network then get yourself a copy
    of ethereal (free at <www.ethereal.com>). It may help you focuss on the real
    problem. If it turns out that you /are/ sending tiny packets then it's
    probably a bug in your code for setting up the buffering. If not then you will
    be able to look elsewhere without distracting yourself with packet sizes.

    -- chris
    Chris Uppal, May 12, 2004
    #16
  17. Parker Thompson

    Yu Song Guest

    Knute Johnson wrote:
    > Roedy Green wrote:
    >
    >> On Tue, 11 May 2004 09:22:29 +0100, Yu Song <> wrote or
    >> quoted :
    >>
    >>
    >>> It could be 100 times faster(depends on your connection) if you
    >>> read/write a chunk of data.

    >>
    >>
    >>
    >> There are two levels of overhead, talking to the socket, and talking
    >> to the buffered reader. Each kind of call has overhead. The bigger
    >> your buffer, the fewer calls to the underlying socket layer.
    >> If you are reading a giant chunk of raw bytes, DON'T use a
    >> BufferedInputStream. Use an unbuffered InputStream, and do it all in
    >> one giant I/O for maximum efficiency.
    >>
    >> see http://mindprod.com/jgloss/buffer.html for other hints on
    >> buffering.

    >
    >
    > Roedy:
    >
    > I didn't think that unbuffered would be as good as buffered so I wrote a
    > couple of test programs and found that reading a whole array at a time
    > was faster than single bytes but that buffering single byte reads was
    > almost as good as buffered or multibyte. The difference between
    > buffered and unbuffered multibyte reads was small but unbuffered was
    > definitely faster.
    >
    > Here are my test programs.
    >
    > import java.io.*;
    > import java.net.*;
    > import java.util.*;
    >
    > public class TestServer {
    > public static void main(String[] args) {
    > byte[] b = new byte[1000];
    > Random r = new Random();
    > while (true) {
    > try {
    > ServerSocket ss = new ServerSocket(8192);
    > System.out.println("Waiting for a connection");
    > Socket s = ss.accept();
    > System.out.println("Connected to: " +
    > s.getInetAddress().getHostAddress());
    > ss.close();
    > OutputStream os = s.getOutputStream();
    > BufferedOutputStream bos = new BufferedOutputStream(os);
    > for (int i=0; i<5000; i++) {
    > r.nextBytes(b);
    > bos.write(b,0,b.length);
    > }
    > os.close();
    > s.close();
    > } catch (IOException ioe) {
    > System.out.println(ioe);
    > }
    > System.out.println("Done\n");
    > }
    > }
    > }
    >



    Because you know the exact data size you will receive.

    Try this client code below (similar to yours but the size of buffer is 512)

    //TestClient.java
    import java.io.*;
    import java.net.*;

    public class TestClient {
    public static void main(String[] args) {
    try {
    byte[] buf = new byte[512];
    int b;
    long start,stop;
    String host = (String)args[0];
    int port = Integer.parseInt(args[1]);
    Socket s = new Socket(host,port);
    InputStream is = s.getInputStream();
    System.out.println("Connected");
    BufferedInputStream bis = new BufferedInputStream(is);
    start = System.currentTimeMillis();
    while ((b = bis.read(buf)) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("Buffered, 512 bytes at a time: " +
    (stop - start) / 1000.0);
    bis.close();
    s.close();
    System.out.println("Done\n");

    s = new Socket(host,port);
    System.out.println("Connected");
    is = s.getInputStream();
    start = System.currentTimeMillis();
    while ((b = is.read(buf)) != -1) ;
    stop = System.currentTimeMillis();
    System.out.println("UnBuffered, 512 bytes at a time: " +
    (stop - start) / 1000.0);
    is.close();
    s.close();
    System.out.println("Done\n");
    } catch (Exception e) {
    System.out.println(e);
    }
    }
    }


    --
    Song

    More info.:
    http://www.dcs.warwick.ac.uk/~esubbn/
    Yu Song, May 12, 2004
    #17
  18. Parker Thompson

    Nigel Wade Guest

    Parker Thompson wrote:
    > Hello,
    >
    > I am trying to send files over the network using tcp sockets and am
    > getting horrible speeds even over fast connections, perhaps 10k/sec where
    > I would expect at least a factor of 10 greater. I feel like this should
    > be pretty simple, but can't seem to figure it out.
    >


    What's your network bandwidth, and what bandwidht are you actually seeing?

    Also, what else is using the network? You might be competing with Sasser ;-)

    --
    Nigel Wade, System Administrator, Space Plasma Physics Group,
    University of Leicester, Leicester, LE1 7RH, UK
    E-mail :
    Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
    Nigel Wade, May 12, 2004
    #18
  19. Roedy Green wrote:
    > On Wed, 12 May 2004 01:10:50 GMT, Knute Johnson
    > <> wrote or quoted :
    >
    >
    >>I didn't think that unbuffered would be as good as buffered so I wrote a
    >>couple of test programs and found that reading a whole array at a time
    >>was faster than single bytes but that buffering single byte reads was
    >>almost as good as buffered or multibyte. The difference between
    >>buffered and unbuffered multibyte reads was small but unbuffered was
    >>definitely faster.

    >
    >
    > That sort of test you must run over and over and average the results.
    >
    > I'm surprised at your result. How many times did you run the test?
    >


    Roedy:

    Here are the results in seconds for 500,000,000 bytes on my local
    100mbit network. I think bigger arrays are probably slightly faster too.

    C:\JavaProjects\widgets>java TestClient 192.168.3.3 8192
    Connected
    Unbuffered, one byte at a time: 1769.469
    Done

    Connected
    Buffered, one byte at a time: 43.078
    Done

    Connected
    Buffered, 1000 bytes at a time: 42.563
    Done

    Connected
    UnBuffered, 1000 bytes at a time: 42.719
    Done


    C:\JavaProjects\widgets>

    --

    Knute Johnson
    email s/nospam/knute/
    Molon labe...
    Knute Johnson, May 12, 2004
    #19
  20. Yu Song wrote:
    > Knute Johnson wrote:
    >
    >> Roedy Green wrote:
    >>
    >>> On Tue, 11 May 2004 09:22:29 +0100, Yu Song <> wrote or
    >>> quoted :
    >>>
    >>>
    >>>> It could be 100 times faster(depends on your connection) if you
    >>>> read/write a chunk of data.
    >>>
    >>>
    >>>
    >>>
    >>> There are two levels of overhead, talking to the socket, and talking
    >>> to the buffered reader. Each kind of call has overhead. The bigger
    >>> your buffer, the fewer calls to the underlying socket layer. If you
    >>> are reading a giant chunk of raw bytes, DON'T use a
    >>> BufferedInputStream. Use an unbuffered InputStream, and do it all in
    >>> one giant I/O for maximum efficiency.
    >>>
    >>> see http://mindprod.com/jgloss/buffer.html for other hints on
    >>> buffering.

    >>
    >>
    >>
    >> Roedy:
    >>
    >> I didn't think that unbuffered would be as good as buffered so I wrote
    >> a couple of test programs and found that reading a whole array at a
    >> time was faster than single bytes but that buffering single byte reads
    >> was almost as good as buffered or multibyte. The difference between
    >> buffered and unbuffered multibyte reads was small but unbuffered was
    >> definitely faster.
    >>
    >> Here are my test programs.
    >>
    >> import java.io.*;
    >> import java.net.*;
    >> import java.util.*;
    >>
    >> public class TestServer {
    >> public static void main(String[] args) {
    >> byte[] b = new byte[1000];
    >> Random r = new Random();
    >> while (true) {
    >> try {
    >> ServerSocket ss = new ServerSocket(8192);
    >> System.out.println("Waiting for a connection");
    >> Socket s = ss.accept();
    >> System.out.println("Connected to: " +
    >> s.getInetAddress().getHostAddress());
    >> ss.close();
    >> OutputStream os = s.getOutputStream();
    >> BufferedOutputStream bos = new BufferedOutputStream(os);
    >> for (int i=0; i<5000; i++) {
    >> r.nextBytes(b);
    >> bos.write(b,0,b.length);
    >> }
    >> os.close();
    >> s.close();
    >> } catch (IOException ioe) {
    >> System.out.println(ioe);
    >> }
    >> System.out.println("Done\n");
    >> }
    >> }
    >> }
    >>

    >
    >
    > Because you know the exact data size you will receive.
    >
    > Try this client code below (similar to yours but the size of buffer is 512)
    >
    > //TestClient.java
    > import java.io.*;
    > import java.net.*;
    >
    > public class TestClient {
    > public static void main(String[] args) {
    > try {
    > byte[] buf = new byte[512];
    > int b;
    > long start,stop;
    > String host = (String)args[0];
    > int port = Integer.parseInt(args[1]);
    > Socket s = new Socket(host,port);
    > InputStream is = s.getInputStream();
    > System.out.println("Connected");
    > BufferedInputStream bis = new BufferedInputStream(is);
    > start = System.currentTimeMillis();
    > while ((b = bis.read(buf)) != -1) ;
    > stop = System.currentTimeMillis();
    > System.out.println("Buffered, 512 bytes at a time: " +
    > (stop - start) / 1000.0);
    > bis.close();
    > s.close();
    > System.out.println("Done\n");
    >
    > s = new Socket(host,port);
    > System.out.println("Connected");
    > is = s.getInputStream();
    > start = System.currentTimeMillis();
    > while ((b = is.read(buf)) != -1) ;
    > stop = System.currentTimeMillis();
    > System.out.println("UnBuffered, 512 bytes at a time: " +
    > (stop - start) / 1000.0);
    > is.close();
    > s.close();
    > System.out.println("Done\n");
    > } catch (Exception e) {
    > System.out.println(e);
    > }
    > }
    > }
    >
    >


    Song:

    Here are the results with your version of TestClient. Almost identical
    to the 1000 byte buffer. Did you get a larger difference?

    C:\JavaProjects\widgets>java SongTestClient 192.168.3.3 8192
    Connected
    Buffered, 512 bytes at a time: 42.656
    Done

    Connected
    UnBuffered, 512 bytes at a time: 42.672
    Done


    C:\JavaProjects\widgets>

    --

    Knute Johnson
    email s/nospam/knute/
    Molon labe...
    Knute Johnson, May 12, 2004
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kal
    Replies:
    1
    Views:
    9,519
    Kevin Spencer
    Jun 21, 2004
  2. Jane Davis

    Network Service account over network

    Jane Davis, Jun 22, 2005, in forum: ASP .Net
    Replies:
    1
    Views:
    440
    Kevin Spencer
    Jun 22, 2005
  3. Bill Volk
    Replies:
    1
    Views:
    3,144
    Bill Volk
    Jul 2, 2003
  4. kin
    Replies:
    0
    Views:
    914
  5. king
    Replies:
    1
    Views:
    274
Loading...

Share This Page