what happens to buffer ?

Discussion in 'Java' started by gk, Mar 30, 2006.

  1. gk

    gk Guest

    byte[] buffer = new byte[512];
    int read;
    while ((read=in.read(buffer)) >0) {
    out.write(buffer, 0, read);
    }


    in the first iteration, the buffer is filled up with bytes.

    what happens in the next iteration ?

    does buffer first cleared off and then filled up afresh

    OR

    the buffer is overwriiten with the new incoming bytes ?
    gk, Mar 30, 2006
    #1
    1. Advertising

  2. gk wrote:
    > byte[] buffer = new byte[512];
    > int read;
    > while ((read=in.read(buffer)) >0) {
    > out.write(buffer, 0, read);
    > }
    >
    >
    > in the first iteration, the buffer is filled up with bytes.
    >
    > what happens in the next iteration ?
    >
    > does buffer first cleared off and then filled up afresh
    >
    > OR
    >
    > the buffer is overwriiten with the new incoming bytes ?
    >


    If "in" is an InputStream reference, the InputStream javadoc covers this
    in detail.

    Patricia
    Patricia Shanahan, Mar 30, 2006
    #2
    1. Advertising

  3. gk

    gk Guest

    > If "in" is an InputStream
    Yes. you are right.

    javadoc says


    "public int read(byte[] b)
    throws IOException

    Reads some number of bytes from the input stream and stores them
    into the buffer array b. The number of bytes actually read is returned
    as an integer. This method blocks until input data is available, end of
    file is detected, or an exception is thrown.

    If b is null, a NullPointerException is thrown. If the length of b
    is zero, then no bytes are read and 0 is returned; otherwise, there is
    an attempt to read at least one byte. If no byte is available because
    the stream is at end of file, the value -1 is returned; otherwise, at
    least one byte is read and stored into b.

    The first byte read is stored into element b[0], the next one into
    b[1], and so on. The number of bytes read is, at most, equal to the
    length of b. Let k be the number of bytes actually read; these bytes
    will be stored in elements b[0] through b[k-1], leaving elements b[k]
    through b[b.length-1] unaffected.

    If the first byte cannot be read for any reason other than end of
    file, then an IOException is thrown. In particular, an IOException is
    thrown if the input stream has been closed.

    The read(b) method for class InputStream has the same effect as:

    read(b, 0, b.length)

    "






    my question is

    what happens in the next iteration ?

    does buffer first cleared off and then filled up afresh

    OR

    the buffer is overwriiten with the new incoming bytes ?


    javadoc does not answer this question.
    gk, Mar 30, 2006
    #3
  4. gk wrote:
    >>If "in" is an InputStream

    >
    > Yes. you are right.
    >
    > javadoc says
    >
    >
    > "public int read(byte[] b)
    > throws IOException
    >
    > Reads some number of bytes from the input stream and stores them
    > into the buffer array b. The number of bytes actually read is returned
    > as an integer. This method blocks until input data is available, end of
    > file is detected, or an exception is thrown.
    >
    > If b is null, a NullPointerException is thrown. If the length of b
    > is zero, then no bytes are read and 0 is returned; otherwise, there is
    > an attempt to read at least one byte. If no byte is available because
    > the stream is at end of file, the value -1 is returned; otherwise, at
    > least one byte is read and stored into b.
    >
    > The first byte read is stored into element b[0], the next one into
    > b[1], and so on. The number of bytes read is, at most, equal to the
    > length of b. Let k be the number of bytes actually read; these bytes
    > will be stored in elements b[0] through b[k-1], leaving elements b[k]
    > through b[b.length-1] unaffected.
    >
    > If the first byte cannot be read for any reason other than end of
    > file, then an IOException is thrown. In particular, an IOException is
    > thrown if the input stream has been closed.
    >
    > The read(b) method for class InputStream has the same effect as:
    >
    > read(b, 0, b.length)
    >
    > "
    >
    >
    >
    >
    >
    >
    > my question is
    >
    > what happens in the next iteration ?
    >
    > does buffer first cleared off and then filled up afresh
    >
    > OR
    >
    > the buffer is overwriiten with the new incoming bytes ?
    >
    >
    > javadoc does not answer this question.
    >


    Yes it does, because it does not say "This is the behavior for the first
    iteration only". The material you quoted applies to every call to
    InputStream's read with a byte buffer, regardless of whether it is the
    first call with that buffer or not.

    Patricia
    Patricia Shanahan, Mar 30, 2006
    #4
  5. gk

    Roedy Green Guest

    On 29 Mar 2006 21:34:25 -0800, "gk" <> wrote, quoted
    or indirectly quoted someone who said :

    >in the first iteration, the buffer is filled up with bytes.
    >
    >what happens in the next iteration ?
    >
    >does buffer first cleared off and then filled up afresh
    >
    >OR
    >
    >the buffer is overwriiten with the new incoming bytes ?


    why would it matter? You know how many bytes there are when you are
    done. If you are just curious, have a look at SRC..ZIP and failing
    that the sun source codes. See http://mindprod.com/jgloss/jdk.html
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, Mar 30, 2006
    #5
  6. gk wrote:
    > byte[] buffer


    a byte array reference "buffer"

    > byte[] buffer = new byte[512];


    is assigned to reference a (new) byte array of 512 bytes.
    Initially these bytes will all contain 0 (ascii NUL).

    > int read;
    > while ((read=in.read(buffer)) >0) {


    here between 1 and 512 bytes of
    the byte array referenced by "buffer"
    are "filled" with byte values
    starting from - the start of the byte array.

    > out.write(buffer, 0, read);
    > }
    >
    >
    > in the first iteration, the buffer is filled up with bytes.
    >
    > what happens in the next iteration ?
    >
    > is buffer first cleared off and then filled up afresh
    >
    > OR
    >
    > the buffer is overwritten with the new incoming bytes ?


    between 1 and 512 bytes of
    the byte array referenced by "buffer"
    are "filled" with byte values
    starting from - the start of the byte array.


    Maybe you read about nio ByteBuffer
    and you are confusing the two?
    Thomas Schodt, Mar 30, 2006
    #6
  7. gk

    gk Guest

    more confused with those answers.

    here i am explaining the problem in a nice way

    say, in the first iteration ,there was 10 bytes in the stream.(because
    its streaming and bytes might come slowly slowly)

    so 10 bytes is read by the read() method and going to the buffer.

    write method uses this byte buffer.



    Now, in the second iteration say, there is 25 bytes in the stream

    so, so 25 bytes is read by the read() method and going to the
    buffer.

    but in the first iteration buffer had 10 bytes ....what will happen
    to those 10 bytes now ?

    does those will be cleared off first and then 25 bytes would be placed.


    OR

    the buffer would be completely overwritten with these new coming 25
    bytes ?







    Thomas Schodt wrote:
    > gk wrote:
    > > byte[] buffer

    >
    > a byte array reference "buffer"
    >
    > > byte[] buffer = new byte[512];

    >
    > is assigned to reference a (new) byte array of 512 bytes.
    > Initially these bytes will all contain 0 (ascii NUL).
    >
    > > int read;
    > > while ((read=in.read(buffer)) >0) {

    >
    > here between 1 and 512 bytes of
    > the byte array referenced by "buffer"
    > are "filled" with byte values
    > starting from - the start of the byte array.so
    >
    > > out.write(buffer, 0, read);
    > > }
    > >
    > >
    > > in the first iteration, the buffer is filled up with bytes.
    > >
    > > what happens in the next iteration ?
    > >
    > > is buffer first cleared off and then filled up afresh
    > >
    > > OR
    > >
    > > the buffer is overwritten with the new incoming bytes ?

    >
    > between 1 and 512 bytes of
    > the byte array referenced by "buffer"
    > are "filled" with byte values
    > starting from - the start of the byte array.
    >
    >
    > Maybe you read about nio ByteBuffer
    > and you are confusing the two?
    gk, Mar 30, 2006
    #7
  8. gk

    Chris Uppal Guest

    gk wrote:

    > the buffer would be completely overwritten with these new coming 25
    > bytes ?


    That's correct. The first 25 bytes of the buffer would be overwritten. The 10
    bytes from the previous read() would be lost.

    (How could the second call to read() "know" that a previous call had put 10
    bytes into the buffer ? And even if it did know, why should it care?
    Presumably if the programmer hadn't wanted to overwrite the existing data, then
    s/he would have used the longer form of read() which takes an argument to say
    where in the buffer to start writing.)

    -- chris
    Chris Uppal, Mar 30, 2006
    #8
  9. gk

    gk Guest

    what should be the size of buffer ?

    is byte[] buffer = new byte[512]; ENOUGH ?

    suppose, at some point of time huge number of bytes (say 1000 bytes)
    stormed .

    Then what will happen ? the buffer cant accept more than 512 bytes
    ......will the additional bytes 1000-512 = 488 will still be in the
    stream ? or they will be lost ?


    the reason is, some people use

    byte[] buffer = new byte[256];
    byte[] buffer = new byte[512];
    byte[] buffer = new byte[1024];

    which one is good ?

    or anything is ok . does it matter really ? does the coder
    responsible for choosing the size of the byte ?
    gk, Mar 30, 2006
    #9
  10. gk wrote (edited):

    > in the first iteration, there were 10 byte values in the stream.
    > (because its streaming and bytes might come slowly slowly)
    >
    > so 10 byte values are read by the read() method and these 10 byte
    > values are stored in the first 10 bytes of the byte array referenced
    > by the byte array reference "buffer".
    >
    > write method uses this byte array.
    >
    >
    >
    > Now, in the second iteration say, there are 25 byte values in the
    > stream
    >
    > so 25 byte values are read by the read() method and these 25 byte
    > values are stored in the first 25 bytes of the byte array referenced
    > by the byte array reference "buffer".
    >
    > but in the first iteration
    > 10 byte values were stored in the byte array.
    > what will happen to those 10 byte values now ?


    They are lost.

    > are those cleared first and then 25 byte values would be placed.
    >
    > OR
    >
    > the buffer would be completely overwritten with these new coming 25
    > bytes ?


    What is the difference?
    Is there a difference? Not that I know of.

    Re 'clear' - Are you asking if read() first stores zeroes in all the
    bytes of the byte array?
    No, it does not.

    Re 'new bytes' - Are you asking if the byte array referenced by "buffer"
    is replaced with a new byte array?
    No, it is not. "buffer" still references the same byte array only now
    some of the bytes of the byte array have new values.

    What are you asking?
    Thomas Schodt, Mar 30, 2006
    #10
  11. On 30 Mar 2006 02:30:31 -0800, gk wrote:
    > what should be the size of buffer ?
    >
    > is byte[] buffer = new byte[512]; ENOUGH ?
    >
    > suppose, at some point of time huge number of bytes (say 1000 bytes)
    > stormed .
    >
    > Then what will happen ? the buffer cant accept more than 512 bytes
    > .....will the additional bytes 1000-512 = 488 will still be in the
    > stream ? or they will be lost ?
    >
    > the reason is, some people use
    >
    > byte[] buffer = new byte[256];
    > byte[] buffer = new byte[512];
    > byte[] buffer = new byte[1024];
    >
    > which one is good ?
    >
    > or anything is ok . does it matter really ? does the coder
    > responsible for choosing the size of the byte ?


    Each time you call read(), the new bytes are written at the start of
    the buffer unless you tell read() to do otherwise. If there were
    already some data in the buffer from a previous read, it will be
    overwritten with the new data.

    Also, read() will never read more than the number of characters you
    request, or the length of the buffer if you don't specify. Note that
    read() can and often will return *fewer* characters than you request,
    so you need to check the return value.

    Any bytes you don't read will wait nicely in the stream until you
    choose to read them.

    So you can decide to read as much or as little as you want each time,
    and can choose an apropriate buffer size. Normally it's more efficient
    to read a lot of data each time and in powers of two, but depending on
    your application you may want to read less.

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, Mar 30, 2006
    #11
  12. gk

    Chris Uppal Guest

    gk wrote:

    > Then what will happen ? the buffer cant accept more than 512 bytes
    > .....will the additional bytes 1000-512 = 488 will still be in the
    > stream ? or they will be lost ?


    The extra bytes remain in the stream until you are ready to read them.


    > the reason is, some people use
    >
    > byte[] buffer = new byte[256];
    > byte[] buffer = new byte[512];
    > byte[] buffer = new byte[1024];
    >
    > which one is good ?


    It doesn't matter very much. In theory the larger the buffer the higher the
    potential speed, but in practise I just choose a number like 4096 and don't
    worry about it.

    The reason it can be faster is that IF each call to read() ends up calling the
    similar function in the underlying OS, then there's a certain fixed overhead
    per call. So the more data you read in one call, the less the overhead when
    averaged over all the bytes you read.

    Note that that doesn't apply if you are using a BufferedInputStream (or
    something like it) because it does the buffering for you. That way you can
    read tiny little chunks at a time (or even single bytes at a time) with very
    little effect on performance.

    -- chris
    Chris Uppal, Mar 30, 2006
    #12
  13. gk wrote:
    > more confused with those answers.
    >
    > here i am explaining the problem in a nice way
    >
    > say, in the first iteration ,there was 10 bytes in the stream.(because
    > its streaming and bytes might come slowly slowly)
    >
    > so 10 bytes is read by the read() method and going to the buffer.
    >
    > write method uses this byte buffer.
    >


    "The first byte read is stored into element b[0], the next one into
    b[1], and so on. The number of bytes read is, at most, equal to the
    length of b. Let k be the number of bytes actually read; these bytes
    will be stored in elements b[0] through b[k-1], leaving elements b[k]
    through b[b.length-1] unaffected."

    Assume all 10 bytes are read by the first call, k is 10 and b is your
    byte buffer. Following the read call, elements 0 through 9 of your
    buffer contain the 10 bytes of read data. Elements 10 through 511 still
    contain whatever they contained before the read call.

    > Now, in the second iteration say, there is 25 bytes in the stream
    >
    > so, so 25 bytes is read by the read() method and going to the
    > buffer.


    "The first byte read is stored into element b[0], the next one into
    b[1], and so on. The number of bytes read is, at most, equal to the
    length of b. Let k be the number of bytes actually read; these bytes
    will be stored in elements b[0] through b[k-1], leaving elements b[k]
    through b[b.length-1] unaffected."

    Assume all bytes are read by the second call, k is 25 and b is your
    byte buffer. Following the read call, elements 0 through 24 of your
    buffer contain the 25 bytes of read data. Elements 25 through 511 still
    contain whatever they contained before the read call.

    The repetition of the quote is deliberate. That paragraph tells you what
    happens to each element of your buffer, regardless of whether it is the
    first read call or the millionth read call.

    > but in the first iteration buffer had 10 bytes ....what will happen
    > to those 10 bytes now ?


    Suppose you had written:

    byte[0] = 7;

    followed some time later by

    byte[0] = 23;

    What happens to the 7? That is what happens to the first 10 elements,
    when you do a read that gets at least 10 bytes of data.

    >
    > does those will be cleared off first and then 25 bytes would be placed.
    >


    I don't see any reason for a prior clear operation, rather than just
    writing the new data over the old.

    Patricia
    Patricia Shanahan, Mar 30, 2006
    #13
  14. Gordon Beaton wrote:
    ....
    > So you can decide to read as much or as little as you want each time,
    > and can choose an apropriate buffer size. Normally it's more efficient
    > to read a lot of data each time and in powers of two, but depending on
    > your application you may want to read less.


    Why the preference for powers of two?

    Patricia
    Patricia Shanahan, Mar 30, 2006
    #14
  15. "Patricia Shanahan" <> wrote in message
    news:klSWf.6445$...
    > Gordon Beaton wrote:
    > ...
    >> So you can decide to read as much or as little as you want each time,
    >> and can choose an apropriate buffer size. Normally it's more efficient
    >> to read a lot of data each time and in powers of two, but depending on
    >> your application you may want to read less.

    >
    > Why the preference for powers of two?
    >
    > Patricia


    Because it looks cool i think, but other than that i can think of exactly
    zero reasons to make buffers a power of 2.
    Remon van Vliet, Mar 30, 2006
    #15
  16. "Gordon Beaton" <> wrote in message
    news:442bf934$...
    > On Thu, 30 Mar 2006 14:57:20 GMT, Patricia Shanahan wrote:
    >> Why the preference for powers of two?

    >
    > When reading from a stream that maps to a file, I believe that read
    > efficiency is improved by aligning reads to OS buffer sizes and
    > ultimately file system or NFS block sizes. AFAIK all of these are
    > normally powers of 2.
    >
    > I suppose if you're reading from a TCP stream, then multiples of MSS
    > bytes might be more appropriate.
    >
    > Superstition? Maybe.
    >
    > /gordon
    >
    > --
    > [ do not email me copies of your followups ]
    > g o r d o n + n e w s @ b a l d e r 1 3 . s e


    Even if that were so, you'd need to know the actual buffer sizes of said OS
    to have a noticable improvement though, and there's a fair chance even then
    the difference is negligable. It's often more useful to adjust the buffer
    size to something sensible for said application. All that said, i always
    allocate general purpose buffers to be a power of 2...there's something
    appealing to the numbers 512 and 4096....maybe i'm just weird
    Remon van Vliet, Mar 30, 2006
    #16
  17. On Thu, 30 Mar 2006 14:57:20 GMT, Patricia Shanahan wrote:
    > Why the preference for powers of two?


    When reading from a stream that maps to a file, I believe that read
    efficiency is improved by aligning reads to OS buffer sizes and
    ultimately file system or NFS block sizes. AFAIK all of these are
    normally powers of 2.

    I suppose if you're reading from a TCP stream, then multiples of MSS
    bytes might be more appropriate.

    Superstition? Maybe.

    /gordon

    --
    [ do not email me copies of your followups ]
    g o r d o n + n e w s @ b a l d e r 1 3 . s e
    Gordon Beaton, Mar 30, 2006
    #17
  18. gk

    Oliver Wong Guest

    [post re-ordered]

    "gk" <> wrote in message
    news:...
    > my question is
    >
    > what happens in the next iteration ?
    >
    > does buffer first cleared off and then filled up afresh
    >
    > OR
    >
    > the buffer is overwriiten with the new incoming bytes ?
    >
    >
    > javadoc does not answer this question.
    >


    It does:

    > Let k be the number of bytes actually read; these bytes
    > will be stored in elements b[0] through b[k-1], leaving elements b[k]
    > through b[b.length-1] unaffected.


    - Oliver
    Oliver Wong, Mar 30, 2006
    #18
  19. gk

    Oliver Wong Guest

    "Patricia Shanahan" <> wrote in message
    news:klSWf.6445$...
    > Gordon Beaton wrote:
    > ...
    >> So you can decide to read as much or as little as you want each time,
    >> and can choose an apropriate buffer size. Normally it's more efficient
    >> to read a lot of data each time and in powers of two, but depending on
    >> your application you may want to read less.

    >
    > Why the preference for powers of two?


    Because it has always been done that way. Do not question your elders!

    - Oliver
    Oliver Wong, Mar 30, 2006
    #19
  20. gk

    Roedy Green Guest

    On Thu, 30 Mar 2006 14:57:20 GMT, Patricia Shanahan <>
    wrote, quoted or indirectly quoted someone who said :

    >> So you can decide to read as much or as little as you want each time,
    >> and can choose an apropriate buffer size. Normally it's more efficient
    >> to read a lot of data each time and in powers of two, but depending on
    >> your application you may want to read less.

    >
    >Why the preference for powers of two?


    Physical i/o is done in terms of some power of two, often 512 bytes If
    your buffer is a nice muliple, physical i/o can do direct to it.
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, Mar 30, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Raja
    Replies:
    12
    Views:
    24,391
    John Harrison
    Jun 21, 2004
  2. NM
    Replies:
    6
    Views:
    464
    Default User
    Sep 20, 2006
  3. Replies:
    2
    Views:
    603
    sergejusz
    Mar 26, 2007
  4. Neal Becker

    buffer creates only read-only buffer?

    Neal Becker, Jan 8, 2009, in forum: Python
    Replies:
    0
    Views:
    412
    Neal Becker
    Jan 8, 2009
  5. xingye
    Replies:
    9
    Views:
    273
    Michael Lu
    Apr 19, 2004
Loading...

Share This Page