how to improve transfering file's efficiency

Discussion in 'Java' started by JTL.zheng, Nov 19, 2006.

  1. JTL.zheng

    JTL.zheng Guest

    I have write program for client to client file transmission
    but it seems unefficient ,the transmission speed is low
    and it take too much CPU resource

    here are the codes(sender's and receiver's)
    are there some ways to improve it about the transfer speed and cpu?


    sender's code:
    -------------------------------------------------------
    input_file = new BufferedInputStream(new FileInputStream(file));
    output_file = new
    BufferedOutputStream(socket_file.getOutputStream());


    byte[] buffer = new byte[10240];
    int readed;
    while ( (readed = input_file.read(buffer)) != -1) {
    output_file.write(buffer, 0, readed);
    }


    output_file.flush();
    -------------------------------------------------------


    receiver's code:
    -------------------------------------------------------
    output_file = new BufferedOutputStream(new
    FileOutputStream(file));
    input_file = new
    BufferedInputStream(socket_file.getInputStream());


    byte[] buffer = new byte[10240];
    int readed;
    while ( (readed = input_file.read(buffer)) != -1) {
    output_file.write(buffer, 0, readed);
    }


    output_file.flush();
    -------------------------------------------------------


    Thank you very much in advance
     
    JTL.zheng, Nov 19, 2006
    #1
    1. Advertising

  2. JTL.zheng

    ChrisWSU Guest

    this is over networking? NIO could potentially increase speed, using
    directly allocated buffers and such.

    Id suggest decreasing the size of the buffer, reading it all at once
    and sending all at once wont necessarily be more efficient because some
    of the data could be being transfered on the wire while your doing the
    reads from disk. generally the issue would be more about your network
    speeds then the java code.

    also you could try wraping it in a gzip stream to decrease the amount
    being sent over the bottleneck (the network), although that increase
    the load on the CPU.
     
    ChrisWSU, Nov 19, 2006
    #2
    1. Advertising

  3. JTL.zheng

    JTL.zheng Guest

    Thank you very much for your help

    yes,it's over networking.
    I have test to decrease the buffer
    10240 and 1024 is almost the same speed
    but if decrease to 10, the speed will slow to 2/3

    the file transfer speed is much lower than my network(LAN)
    so I think maybe it could get a higher speed

    >also you could try wraping it in a gzip stream to decrease the amount


    what's a "gzip stream"?
     
    JTL.zheng, Nov 21, 2006
    #3
  4. JTL.zheng

    Guest

    JTL.zheng a écrit :

    > Thank you very much for your help
    >
    > yes,it's over networking.
    > I have test to decrease the buffer
    > 10240 and 1024 is almost the same speed
    > but if decrease to 10, the speed will slow to 2/3
    >
    > the file transfer speed is much lower than my network(LAN)
    > so I think maybe it could get a higher speed
    >
    > >also you could try wraping it in a gzip stream to decrease the amount

    >
    > what's a "gzip stream"?


    If you use a byte array to read (or write) in your stream, you don't
    need to use a BufferedStream.

    A simple stream is better, because in this case a BufferedStream must
    copy your buffer in his self buffer, a not necessary operation
     
    , Nov 21, 2006
    #4
  5. JTL.zheng

    Chris Uppal Guest

    JTL.zheng wrote:

    > the file transfer speed is much lower than my network(LAN)
    > so I think maybe it could get a higher speed


    How fast is your LAN ? How fast is your code running ?

    Also, do you have any other software which can transfer data between those two
    machines over the same LAN at a noticeably faster rate ? It may be a problem
    with the LAN itself rather than either your code or the Java platform.

    I can't see anything much wrong with the code, so I'd expect it to pretty much
    saturate either the LAN itself or the network cards on one or the other
    machine. I would /not/ expect it to max out the CPU.

    It might be worth getting rid of the buffering around the InputStream. I don't
    think that'll make much difference, but it may be worth the trial. Since you
    are using a large read buffer anyway, the extra buffering will only incur extra
    copying costs. I would leave the buffers on the output side in place, so that
    if you do see short reads (just a few bytes at a time), they won't translate
    into short writes too.

    Another thing that might be worth trying is to remove the write()s in the
    sender, and see how fast that can run when all it's doing is reading the file.
    Similarly, remove the writes() in the reciever and see how fast that can run
    when it is only reading from the LAN and not writing a file too.

    (I would also, by now, have fired up Ethereal[*] to see if there was anything
    odd happening at the network level -- very small packets, or lots of retries,
    or something of that kind.)

    -- chris

    [*] Apparently now also known as "Wireshark" -- silly name.
     
    Chris Uppal, Nov 21, 2006
    #5
  6. JTL.zheng

    JTL.zheng Guest

    Thank you very much for yours advices

    I have test several modes(have/not have I/O Buffere) and the size of
    the buffer
    and get rid of the writing file in recever when test
    now the speed is avg. 10.2MB/s (my LAN is 100Mbps)

    my new codes is:
    --------------------------------
    input_file = new FileInputStream(file);
    output_file = socket_file.getOutputStream();

    byte[] buffer = new byte[2048000];
    int readed;
    while ( (readed = input_file.read(buffer)) != -1) {
    output_file.write(buffer, 0, readed);
    }
    --------------------------------
    I get rid of the BufferedInput/BufferedOutput as it doesn't improve the
    transfer speed
    and take more CPU time.

    I also change the size of the buffer
    when 1KB,the speed is 5.4MB/s
    10KB is 7.7 MB/s
    but after 100KB(500KB,1MB,2MB,5MB,10MB)
    the speed is standing in 10.2MB/s

    and it take less cpu when the buffer is bigger
    after 1MB it stand in 80%

    so my default buffer is 2048000B
    (I also add a JComboBox using to set the buffer size)

    but the cpu is still a litte high
    it is avg. 78% (my cpu is AMD Athlon 2500+)
    is that any way to low down the cpu?
     
    JTL.zheng, Dec 3, 2006
    #6
  7. JTL.zheng

    Chris Uppal Guest

    JTL.zheng wrote:

    > I have test several modes(have/not have I/O Buffere) and the size of
    > the buffer
    > and get rid of the writing file in recever when test
    > now the speed is avg. 10.2MB/s (my LAN is 100Mbps)


    That's reasonable (but see below). Once you allow for the overheads of TCP,
    and IP, you are not going to get very much closer to the theoretical bandwidth
    of 12.5 MB/s (or 11.9 if you use binary megabytes).


    > I also change the size of the buffer
    > when 1KB,the speed is 5.4MB/s
    > 10KB is 7.7 MB/s
    > but after 100KB(500KB,1MB,2MB,5MB,10MB)
    > the speed is standing in 10.2MB/s


    That's a bit odd. In a test on my LAN I don't have to use nearly that big a
    buffer to get reasonable speeds. Using a 1024 byte[] read/write buffer (and no
    BuifferedXxxputStreams) I see speeds of 11.7 MB/s reading and 11.4 writing.
    Increasing the buffer size to (8*1024) bytes results in 11.7 MB/s reading and
    writing. Increasing it again to (64*1024) made no significant difference. (My
    MBs here are one million bytes, not 1024*1024).

    The buffer size made a bigger difference to the CPU load when writing. The CPU
    load was about 75% when reading using any of the three buffer sizes; but when
    writing it was at about 85% for the smaller buffer, around 50% with the 8K
    buffer, and about 45% with the largest buffer.

    I don't know why this machine/network is getting better figures than you from a
    smaller buffer. FWIW, the test machine is a 1.5GHz Celeron laptop running
    WinXP Pro (sp1, not sp2). The other machine in the test is a faster Win2K
    machine with a better network card and using huge buffers (so it shouldn't
    affect the tests at all). The test machine was using the JDK 1.5.0_6 client
    JVM. (-server gives essentially the same numbers). The LAN itself is an
    unloaded 100Mbps Ethernet; there are two switched hubs between the two machines
    (but I know from previous testing that they don't affect bandwidth tests
    measurably).

    One thing that /might/ be relevant is that neither of my test machines use
    Window's built-in firewall/packet filtering, and have no other internal
    fire-wall. I don't /think/ that either machines' virus checkers actively
    monitor the contents of arbitrary TCP streams (I was using ports 3344 and 4433
    for the tests) but it's impossible to be sure.

    -- chris
     
    Chris Uppal, Dec 4, 2006
    #7
  8. JTL.zheng

    JTL.zheng Guest

    Thank you very much for your particular explanation Chris Uppal.
    As it has almost get the limit of the LAN's bandwidth and I have some
    examinations to prepare, I have not modified it these weeks.
    but it is used to transfer files in our LAN now, and it works well.
    Thanks again for all of your help.
    : )
     
    JTL.zheng, Dec 21, 2006
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. David Hilsee
    Replies:
    4
    Views:
    425
    Vaclav Haisman
    Sep 1, 2004
  2. JTL.zheng
    Replies:
    0
    Views:
    279
    JTL.zheng
    Nov 19, 2006
  3. Peng Jian

    Can static variable improve efficiency?

    Peng Jian, Jul 12, 2004, in forum: C Programming
    Replies:
    9
    Views:
    627
  4. shuisheng
    Replies:
    3
    Views:
    385
    tonydee
    Apr 5, 2010
  5. Johnson Lau

    improve the efficiency

    Johnson Lau, Oct 6, 2003, in forum: Perl Misc
    Replies:
    7
    Views:
    169
    Eric Bohlman
    Oct 6, 2003
Loading...

Share This Page