creating byte[] with subfields

Discussion in 'Java' started by Roedy Green, Jan 18, 2013.

  1. Roedy Green

    Roedy Green Guest

    I often have to construct byte arrays with binary fields, or read
    them.

    I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    I wondered if there is a simpler way, one that does not require
    computing each byte individually with shifts.
    --
    Roedy Green Canadian Mind Products http://mindprod.com
    The first 90% of the code accounts for the first 90% of the development time.
    The remaining 10% of the code accounts for the other 90% of the development
    time.
    ~ Tom Cargill Ninety-ninety Law
     
    Roedy Green, Jan 18, 2013
    #1
    1. Advertising

  2. Roedy Green

    Jim Janney Guest

    Roedy Green <> writes:

    > I often have to construct byte arrays with binary fields, or read
    > them.
    >
    > I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    > I wondered if there is a simpler way, one that does not require
    > computing each byte individually with shifts.


    In C, sure. Java by design doesn't allow those kinds of tricks.

    --
    Jim Janney
     
    Jim Janney, Jan 18, 2013
    #2
    1. Advertising

  3. Roedy Green

    Nigel Wade Guest

    On 18/01/13 08:35, Roedy Green wrote:
    > I often have to construct byte arrays with binary fields, or read
    > them.
    >
    > I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    > I wondered if there is a simpler way, one that does not require
    > computing each byte individually with shifts.
    >


    BitSet, and its toByteArray() method may do what you require.

    I've not tried it myself. I still do it by shift/mask.

    --
    Nigel Wade
     
    Nigel Wade, Jan 18, 2013
    #3
  4. Roedy Green

    Jim Janney Guest

    Jim Janney <> writes:

    > Roedy Green <> writes:
    >
    >> I often have to construct byte arrays with binary fields, or read
    >> them.
    >>
    >> I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    >> I wondered if there is a simpler way, one that does not require
    >> computing each byte individually with shifts.

    >
    > In C, sure. Java by design doesn't allow those kinds of tricks.


    The hotspot compiler might notice that you're shifting and masking by 8
    bits and replace it with direct byte addressing. That seems like a
    straightforward optimization. OTOH, the speed difference is probably
    too tiny to notice anyway.

    --
    Jim Janney
     
    Jim Janney, Jan 18, 2013
    #4
  5. Roedy Green

    Daniel Pitts Guest

    On 1/18/13 12:35 AM, Roedy Green wrote:
    > I often have to construct byte arrays with binary fields, or read
    > them.
    >
    > I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    > I wondered if there is a simpler way, one that does not require
    > computing each byte individually with shifts.
    >

    How about using a BitSet instead?

    Or, avoid this level of low-level work?

    Or, create a helper wrapper class. BitStream, where you can write a
    number of bits, rather than whole bytes.
     
    Daniel Pitts, Jan 18, 2013
    #5
  6. Roedy Green

    Roedy Green Guest

    On Fri, 18 Jan 2013 10:53:34 -0800, Daniel Pitts
    <> wrote, quoted or indirectly
    quoted someone who said :

    >How about using a BitSet instead?


    I don't often have to do anything finer than the byte.
    On project I did recently involved XOR encryption, where you need
    everything in bytes to be able to xor.

    Last night I was composing/decomposing a DataPacket for
    SingleInstance.

    I did one to handle Network Time Protocol.
    --
    Roedy Green Canadian Mind Products http://mindprod.com
    The first 90% of the code accounts for the first 90% of the development time.
    The remaining 10% of the code accounts for the other 90% of the development
    time.
    ~ Tom Cargill Ninety-ninety Law
     
    Roedy Green, Jan 18, 2013
    #6
  7. Roedy Green

    Daniel Pitts Guest

    On 1/18/13 2:23 PM, Roedy Green wrote:
    > On Fri, 18 Jan 2013 10:53:34 -0800, Daniel Pitts
    > <> wrote, quoted or indirectly
    > quoted someone who said :
    >
    >> How about using a BitSet instead?

    >
    > I don't often have to do anything finer than the byte.
    > On project I did recently involved XOR encryption, where you need
    > everything in bytes to be able to xor.
    >
    > Last night I was composing/decomposing a DataPacket for
    > SingleInstance.
    >
    > I did one to handle Network Time Protocol.
    >

    FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    encrypt, at least not the naive approach to it.
     
    Daniel Pitts, Jan 18, 2013
    #7
  8. Roedy Green

    Arne Vajhøj Guest

    On 1/18/2013 3:35 AM, Roedy Green wrote:
    > I often have to construct byte arrays with binary fields, or read
    > them.
    >
    > I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    > I wondered if there is a simpler way, one that does not require
    > computing each byte individually with shifts.


    A DataOutputStream on top of a ByteArrayOutputStream is the
    old Java way of doing it. And if you do not need to manipulate bits
    and you are happy with big endian then you should not to use shifts.

    The new Java way is ByteBuffer which support both big and little
    endian, so you only need shift for bit manipulation.

    Those are both pretty low level. If you need something more
    high level, then you will need to look outside of standard
    Java API.

    Arne

    PS: I actually have something on the shelf for that.
     
    Arne Vajhøj, Jan 19, 2013
    #8
  9. Roedy Green

    Arne Vajhøj Guest

    On 1/18/2013 8:33 PM, Arne Vajhøj wrote:
    > On 1/18/2013 3:35 AM, Roedy Green wrote:
    >> I often have to construct byte arrays with binary fields, or read
    >> them.
    >>
    >> I use a ByteArrayOutputStream and a DataOutputStream to compose them.
    >> I wondered if there is a simpler way, one that does not require
    >> computing each byte individually with shifts.

    >
    > A DataOutputStream on top of a ByteArrayOutputStream is the
    > old Java way of doing it. And if you do not need to manipulate bits
    > and you are happy with big endian then you should not to use shifts.
    >
    > The new Java way is ByteBuffer which support both big and little
    > endian, so you only need shift for bit manipulation.
    >
    > Those are both pretty low level. If you need something more
    > high level, then you will need to look outside of standard
    > Java API.


    > PS: I actually have something on the shelf for that.


    Data class:

    import dk.vajhoej.record.Endian;
    import dk.vajhoej.record.FieldType;
    import dk.vajhoej.record.Struct;
    import dk.vajhoej.record.StructField;

    @Struct(endianess=Endian.LITTLE)
    public class Data {
    @StructField(n=0,type=FieldType.BIT,length=4)
    private int bf1;
    @StructField(n=1,type=FieldType.BIT,length=4)
    private int bf2;
    @StructField(n=2,type=FieldType.INT4)
    private int iv;

    @StructField(n=3,type=FieldType.VARSTR,prefixlength=1,encoding="ISO-8859-1")
    private String sv;
    public Data() {
    this(0, 0, 0, "");
    }
    public Data(int bf1, int bf2, int iv, String sv) {
    this.bf1 = bf1;
    this.bf2 = bf2;
    this.iv = iv;
    this.sv = sv;
    }
    public int getBf1() {
    return bf1;
    }
    public void setBf1(int bf1) {
    this.bf1 = bf1;
    }
    public int getBf2() {
    return bf2;
    }
    public void setBf2(int bf2) {
    this.bf2 = bf2;
    }
    public int getIv() {
    return iv;
    }
    public void setIv(int iv) {
    this.iv = iv;
    }
    public String getSv() {
    return sv;
    }
    public void setSv(String sv) {
    this.sv = sv;
    }
    }

    Test program:

    import dk.vajhoej.record.RecordException;
    import dk.vajhoej.record.StructReader;
    import dk.vajhoej.record.StructWriter;

    public class RW {
    public static void main(String[] args) throws RecordException {
    Data o1 = new Data(1, 2, 3, "ABC");
    StructWriter sw = new StructWriter();
    sw.write(o1);
    byte[] ba = sw.getBytes();
    for(byte b1 : ba) {
    System.out.printf(" %02X" , b1);
    }
    System.out.println();
    StructReader sr = new StructReader(ba);
    Data o2 = sr.read(Data.class);
    System.out.println(o2.getBf1() + " " + o2.getBf2() + " " + o2.getIv()
    + " " + o2.getSv());
    }
    }

    Output:

    12 03 00 00 00 03 41 42 43
    1 2 3 ABC

    Arne
     
    Arne Vajhøj, Jan 19, 2013
    #9
  10. Roedy Green

    Roedy Green Guest

    On Fri, 18 Jan 2013 15:16:45 -0800, Daniel Pitts
    <> wrote, quoted or indirectly
    quoted someone who said :

    >FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    >encrypt, at least not the naive approach to it.


    It depends. I just finished a project which I will be permitted to
    release into the public domain in a year which uses one time pads,
    which XOR messages with strings of truly random bits.

    The only way you can crack it is to crack the machine at either end or
    intercept the key transfer. You can't do it just by studying
    messages.
    --
    Roedy Green Canadian Mind Products http://mindprod.com
    The first 90% of the code accounts for the first 90% of the development time.
    The remaining 10% of the code accounts for the other 90% of the development
    time.
    ~ Tom Cargill Ninety-ninety Law
     
    Roedy Green, Jan 19, 2013
    #10
  11. Roedy Green

    Arne Vajhøj Guest

    On 1/18/2013 6:16 PM, Daniel Pitts wrote:
    > FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    > encrypt, at least not the naive approach to it.


    Not today. 200 years ago it would (1 byte XOR is really
    a special case of 1 replacement alphabet and N byte XOR
    of N replacements alphabets).

    Arne
     
    Arne Vajhøj, Jan 19, 2013
    #11
  12. Roedy Green

    Daniel Pitts Guest

    On 1/18/13 11:24 PM, Roedy Green wrote:
    > On Fri, 18 Jan 2013 15:16:45 -0800, Daniel Pitts
    > <> wrote, quoted or indirectly
    > quoted someone who said :
    >
    >> FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    >> encrypt, at least not the naive approach to it.

    >
    > It depends. I just finished a project which I will be permitted to
    > release into the public domain in a year which uses one time pads,
    > which XOR messages with strings of truly random bits.
    >
    > The only way you can crack it is to crack the machine at either end or
    > intercept the key transfer. You can't do it just by studying
    > messages.
    >

    It really does depend.

    If the plain-text are formed poorly (eg. plain ascii text, or contains
    any well known substrings), and the key is sufficiently small (needs to
    be repeated several times) one could easily figure out the key, or at
    least part of it. That could be enough to turn it into a simple word
    game, which will eventually reveal the entire plain-text.
     
    Daniel Pitts, Jan 20, 2013
    #12
  13. On 1/19/2013 11:46 PM, Daniel Pitts wrote:
    > On 1/18/13 11:24 PM, Roedy Green wrote:
    >> On Fri, 18 Jan 2013 15:16:45 -0800, Daniel Pitts
    >> <> wrote, quoted or indirectly
    >> quoted someone who said :
    >>
    >>> FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    >>> encrypt, at least not the naive approach to it.

    >>
    >> It depends. I just finished a project which I will be permitted to
    >> release into the public domain in a year which uses one time pads,
    >> which XOR messages with strings of truly random bits.
    >>
    >> The only way you can crack it is to crack the machine at either end or
    >> intercept the key transfer. You can't do it just by studying
    >> messages.
    >>

    > It really does depend.


    No, it doesn't. A one-time pad is provably 100% perfectly secure, and is
    the only cryptographic method to be so proven. The requirements is that
    you have a string of random bits as long as the message you send and you
    never reuse those random bits ever again.

    --
    Beware of bugs in the above code; I have only proved it correct, not
    tried it. -- Donald E. Knuth
     
    Joshua Cranmer, Jan 20, 2013
    #13
  14. Roedy Green

    Arne Vajhøj Guest

    On 1/20/2013 12:46 AM, Daniel Pitts wrote:
    > On 1/18/13 11:24 PM, Roedy Green wrote:
    >> On Fri, 18 Jan 2013 15:16:45 -0800, Daniel Pitts
    >> <> wrote, quoted or indirectly
    >> quoted someone who said :
    >>
    >>> FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    >>> encrypt, at least not the naive approach to it.

    >>
    >> It depends. I just finished a project which I will be permitted to
    >> release into the public domain in a year which uses one time pads,
    >> which XOR messages with strings of truly random bits.
    >>
    >> The only way you can crack it is to crack the machine at either end or
    >> intercept the key transfer. You can't do it just by studying
    >> messages.
    >>

    > It really does depend.
    >
    > If the plain-text are formed poorly (eg. plain ascii text, or contains
    > any well known substrings), and the key is sufficiently small (needs to
    > be repeated several times)


    A one time pad by definition has a key size equal to message
    size so the key does not repeat.

    Arne
     
    Arne Vajhøj, Jan 20, 2013
    #14
  15. On 1/20/2013 12:01 PM, Joshua Cranmer wrote:
    > On 1/19/2013 11:46 PM, Daniel Pitts wrote:
    >> On 1/18/13 11:24 PM, Roedy Green wrote:
    >>> On Fri, 18 Jan 2013 15:16:45 -0800, Daniel Pitts
    >>> <> wrote, quoted or indirectly
    >>> quoted someone who said :
    >>>
    >>>> FWIW, XOR encryption is more like XOR obfuscation. It doesn't actually
    >>>> encrypt, at least not the naive approach to it.
    >>>
    >>> It depends. I just finished a project which I will be permitted to
    >>> release into the public domain in a year which uses one time pads,
    >>> which XOR messages with strings of truly random bits.
    >>>
    >>> The only way you can crack it is to crack the machine at either end or
    >>> intercept the key transfer. You can't do it just by studying
    >>> messages.
    >>>

    >> It really does depend.

    >
    > No, it doesn't. A one-time pad is provably 100% perfectly secure, and is
    > the only cryptographic method to be so proven. The requirements is that
    > you have a string of random bits as long as the message you send and you
    > never reuse those random bits ever again.


    And there is a good reason why it is not called two times pad.

    In the late forties the Russians had a shortage of one time pads
    and started using twice - once for military stuff and one for
    civilian stuff. In the sixties the British and Americans discovered
    that and started cracking it.

    Arne
     
    Arne Vajhøj, Jan 20, 2013
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andreas
    Replies:
    1
    Views:
    881
    Jonathan Bromley
    May 4, 2004
  2. Bharat Bhushan

    Appending byte[] to another byte[] array

    Bharat Bhushan, Aug 5, 2003, in forum: Java
    Replies:
    15
    Views:
    40,324
    Roedy Green
    Aug 5, 2003
  3. Jean-Daniel Gamache
    Replies:
    0
    Views:
    431
    Jean-Daniel Gamache
    Jul 14, 2004
  4. Peter
    Replies:
    3
    Views:
    735
    Michael Borgwardt
    Aug 5, 2004
  5. Kirby
    Replies:
    3
    Views:
    659
    Kirby
    Oct 8, 2004
Loading...

Share This Page