convert byte stream to integers

Discussion in 'Perl Misc' started by Toralf Förster, Sep 7, 2010.

  1. Toralf Förster, Sep 7, 2010
    #1
    1. Advertising

  2. Ben Morrow wrote:
    > perldoc -f unpack


    Yep, RTFM b/c I made the wrong question, ok again :

    I'm unsure how do I read the 2nd 4byte integer in :

    tfoerste@n22 ~/download $ dd if=cohnopafinal_0.65_sha512_portion04 of=4byte
    bs=4 count=2
    2+0 records in
    2+0 records out
    8 bytes (8 B) copied, 0.000438815 s, 18.2 kB/s

    because this gives only the answer for the first 4 bytes :

    tfoerste@n22 ~/download $ cat 4byte | perl -wane 'print unpack "N", $_;
    print "\n"'
    1831393600



    --
    MfG/Sincerely
    Toralf Förster
    pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
     
    Toralf Förster, Sep 7, 2010
    #2
    1. Advertising

  3. Toralf Förster

    Steve C Guest

    On 09/07/2010 11:03 AM, Toralf Förster wrote:
    > Reading this article
    > http://mpl.mpg.de/mpf/php/abteilung...w=workgroups&in=qiv&and=&page=qiv/quantumbits
    > I'm wondering how can a readin from this file
    > http://mpl.mpg.de/mpf/php/abteilung1/template/qiv/200MiB/cohnopafinal_0.65_sha512_portion04
    > with a perl program - eg. 32bit integers
    >


    much like with any other language (perl functions in parens)

    1) open the file (open)
    2) loop through until end-of-file (while, eof)
    3) read 32 bits from the file (read)
    4) store the data in a variable (unpack)
    5) process it - your code here
    6) when done close the file (close)

    The unpack might seem non-obvious. It takes a string and unpacks it as a perl value.
    The reason this step is separate from the read is that perl variables have to know
    what type of data they contain, since they can hold different types.
     
    Steve C, Sep 7, 2010
    #3
  4. Toralf Förster

    Uri Guttman Guest

    >>>>> "TF" == Toralf Förster <> writes:

    TF> Ben Morrow wrote:
    >> perldoc -f unpack


    TF> Yep, RTFM b/c I made the wrong question, ok again :

    TF> I'm unsure how do I read the 2nd 4byte integer in :

    TF> tfoerste@n22 ~/download $ dd if=cohnopafinal_0.65_sha512_portion04 of=4byte
    TF> bs=4 count=2
    TF> 2+0 records in
    TF> 2+0 records out
    TF> 8 bytes (8 B) copied, 0.000438815 s, 18.2 kB/s

    TF> because this gives only the answer for the first 4 bytes :

    TF> tfoerste@n22 ~/download $ cat 4byte | perl -wane 'print unpack "N", $_;
    TF> print "\n"'
    TF> 1831393600

    ever heard of a loop or a quantifier? N only converts one word so why
    would you think it would 2 words? there is also a perlpacktut doc you
    should read. pack/unpack have lots of formats and options to handle most
    any binary data out there. and if you can't do it in one call, you can
    use perl to grab data for use by pack/unpack.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Sep 7, 2010
    #4
  5. Ben Morrow wrote:
    >...
    > Ben


    Thx Ben for your help, now I got it.

    --
    MfG/Sincerely
    Toralf Förster
    pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
     
    Toralf Förster, Sep 8, 2010
    #5
  6. Steve C wrote:
    > The unpack might seem non-obvious. It takes a string and unpacks it as a
    > perl value. The reason this step is separate from the read is that perl
    > variables have to know what type of data they contain, since they can hold
    > different types.


    Yep thx, I thought it could be done in one step.

    This is now sufficient for me :

    perl -we 'open (FILE, "./4byte"); while (read (FILE, my $s, 16)) { my @N =
    unpack "I*", $s; print join ("\t", @N), "\n"; } close (FILE)'


    --
    MfG/Sincerely
    Toralf Förster
    pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
     
    Toralf Förster, Sep 8, 2010
    #6
  7. Toralf Förster

    Uri Guttman Guest

    >>>>> "TF" == Toralf Förster <> writes:

    TF> Steve C wrote:
    >> The unpack might seem non-obvious. It takes a string and unpacks it as a
    >> perl value. The reason this step is separate from the read is that perl
    >> variables have to know what type of data they contain, since they can hold
    >> different types.


    TF> Yep thx, I thought it could be done in one step.

    it can be. you need to learn perl one liner tricks.

    TF> This is now sufficient for me :

    TF> perl -we 'open (FILE, "./4byte"); while (read (FILE, my $s, 16)) { my @N =
    TF> unpack "I*", $s; print join ("\t", @N), "\n"; } close (FILE)'


    with perl -0777n you can read the whole file into $_. then split to the
    size you want unpack and print. much simpler than your code. there is
    also no need for the close or the temp vars you have.

    uri

    --
    Uri Guttman ------ -------- http://www.sysarch.com --
    ----- Perl Code Review , Architecture, Development, Training, Support ------
    --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
     
    Uri Guttman, Sep 8, 2010
    #7
  8. Toralf Förster

    C.DeRykus Guest

    On Sep 8, 9:44 am, Toralf Förster <> wrote:
    > Steve C wrote:
    > > The unpack might seem non-obvious.  It takes a string and unpacks it as a
    > > perl value. The reason this step is separate from the read is that perl
    > > variables have to know what type of data they contain, since they can hold
    > > different types.

    >
    > Yep thx, I thought it could be done in one step.
    >
    > This is now sufficient for me :
    >
    > perl -we 'open (FILE, "./4byte"); while (read (FILE, my $s, 16)) { my @N =
    > unpack "I*", $s; print join ("\t", @N), "\n"; } close (FILE)'
    >


    That can be simplified a bit without getting golf-ish:


    perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    F,$s,16'

    --
    Charles DeRykus
     
    C.DeRykus, Sep 9, 2010
    #8
  9. C.DeRykus wrote:
    > perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    > F,$s,16'


    You need the -l switch for that to work the same as the OP's

    perl -lne'BEGIN{$/=\16;$,="\t"}print unpack"I*",$_' ./4byte



    John
    --
    Any intelligent fool can make things bigger and
    more complex... It takes a touch of genius -
    and a lot of courage to move in the opposite
    direction. -- Albert Einstein
     
    John W. Krahn, Sep 9, 2010
    #9
  10. Toralf Förster

    Steve C Guest

    On 09/08/2010 12:44 PM, Toralf Förster wrote:
    > Steve C wrote:
    >> The unpack might seem non-obvious. It takes a string and unpacks it as a
    >> perl value. The reason this step is separate from the read is that perl
    >> variables have to know what type of data they contain, since they can hold
    >> different types.

    >
    > Yep thx, I thought it could be done in one step.
    >
    > This is now sufficient for me :
    >
    > perl -we 'open (FILE, "./4byte"); while (read (FILE, my $s, 16)) { my @N =
    > unpack "I*", $s; print join ("\t", @N), "\n"; } close (FILE)'
    >
    >



    I forgot to mention that after opening a binary file, you should call
    binmode on the filehandle to protect the data from unwanted text file
    conversions by the I/O services. Something like this:

    open (FILE, '<', "./4byte") or die "Unable to open file\n";
    binmode FILE;
     
    Steve C, Sep 9, 2010
    #10
  11. Toralf Förster

    Dr.Ruud Guest

    On 2010-09-09 22:20, Steve C wrote:

    > I forgot to mention that after opening a binary file, you should call
    > binmode on the filehandle to protect the data from unwanted text file
    > conversions by the I/O services. Something like this:
    >
    > open (FILE, '<', "./4byte") or die "Unable to open file\n";
    > binmode FILE;


    Or just use the :raw layer, see perlio.

    open my $fh, "<:raw", $fname
    or die "Error with '$fname': ", $!;

    --
    Ruud
     
    Dr.Ruud, Sep 9, 2010
    #11
  12. Toralf Förster

    sreservoir Guest

    On 9/9/2010 3:31 PM, John W. Krahn wrote:
    > C.DeRykus wrote:
    >> perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    >> F,$s,16'

    >
    > You need the -l switch for that to work the same as the OP's
    >
    > perl -lne'BEGIN{$/=\16;$,="\t"}print unpack"I*",$_' ./4byte
    >
    > John


    with 5.10, -E and say.

    INIT does well, and unpack takes implicit $_. and a v-string for $,.

    on the command line ./4byte should be the same thing as 4byte.

    perl -nE'INIT{$/=\16;$,=v9}say unpack"I*"' 4byte # 9 chars shorter.

    --

    "Six by nine. Forty two."
    "That's it. That's all there is."
    "I always thought something was fundamentally wrong with the universe."
     
    sreservoir, Sep 9, 2010
    #12
  13. Toralf Förster

    C.DeRykus Guest

    On Sep 9, 12:31 pm, "John W. Krahn" <> wrote:
    > C.DeRykus wrote:
    > > perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    > > F,$s,16'

    >
    > You need the -l switch for that to work the same as the OP's
    >
    > perl -lne'BEGIN{$/=\16;$,="\t"}print unpack"I*",$_' ./4byte
    >


    Neat. Still, neither is safe on Win32
    without binmode - both input/output.

    perl -lne 'use open IO=>":raw"; ..."

    --
    Charles DeRykus
     
    C.DeRykus, Sep 10, 2010
    #13
  14. Toralf Förster

    sreservoir Guest

    On 9/10/2010 4:07 AM, C.DeRykus wrote:
    > On Sep 9, 12:31 pm, "John W. Krahn"<> wrote:
    >> C.DeRykus wrote:
    >>> perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    >>> F,$s,16'

    >>
    >> You need the -l switch for that to work the same as the OP's
    >>
    >> perl -lne'BEGIN{$/=\16;$,="\t"}print unpack"I*",$_' ./4byte
    >>

    >
    > Neat. Still, neither is safe on Win32
    > without binmode - both input/output.
    >
    > perl -lne 'use open IO=>":raw"; ..."


    no. perl -Mopen=IO,:raw ....

    --

    "Six by nine. Forty two."
    "That's it. That's all there is."
    "I always thought something was fundamentally wrong with the universe."
     
    sreservoir, Sep 10, 2010
    #14
  15. Ben Morrow wrote:
    >
    > Quoth sreservoir<>:
    >> On 9/10/2010 4:07 AM, C.DeRykus wrote:
    >>> On Sep 9, 12:31 pm, "John W. Krahn"<> wrote:
    >>>> C.DeRykus wrote:
    >>>>> perl -we 'open(F,"./4byte"); $,="\t"; print unpack "I*",$s while read
    >>>>> F,$s,16'
    >>>>
    >>>> You need the -l switch for that to work the same as the OP's
    >>>>
    >>>> perl -lne'BEGIN{$/=\16;$,="\t"}print unpack"I*",$_' ./4byte
    >>>>
    >>>
    >>> Neat. Still, neither is safe on Win32
    >>> without binmode - both input/output.
    >>>
    >>> perl -lne 'use open IO=>":raw"; ..."

    >>
    >> no. perl -Mopen=IO,:raw ....

    >
    > No. STD{IN,OUT,ERR} are opened before open.pm is loaded, so it doesn't
    > affect them:
    >
    > ~% PERLIO=:perlio:crlf perl -Mopen=IO,:raw
    > -E'say for PerlIO::get_layers \*STDIN'
    > unix
    > perlio
    > crlf
    > ~%
    >
    > There isn't any way to get open.pm to push any layer other than
    > :encoding or :utf8 onto the STD handles (no, I don't understand why
    > either). I think the best you're going to get is
    >
    > perl -lne'BEGIN{binmode$_ for<STD{IN,OUT}>;...}...'
    >
    > (now *there's* a nasty golf trick...).


    Yes but, Perl doesn't use STDIN with the -n or -p switch, it uses ARGV,
    and if you use the -i switch as well it uses ARGVOUT instead of STDOUT.



    John
    --
    Any intelligent fool can make things bigger and
    more complex... It takes a touch of genius -
    and a lot of courage to move in the opposite
    direction. -- Albert Einstein
     
    John W. Krahn, Sep 11, 2010
    #15
  16. Toralf Förster

    sreservoir Guest

    On 9/10/2010 8:08 PM, Ben Morrow wrote:
    > [snip]
    > perl -lne'BEGIN{binmode$_ for<STD{IN,OUT}>;...}...'
    >
    > (now *there's* a nasty golf trick...).


    INIT!

    --

    "Six by nine. Forty two."
    "That's it. That's all there is."
    "I always thought something was fundamentally wrong with the universe."
     
    sreservoir, Sep 11, 2010
    #16
  17. Toralf Förster

    sreservoir Guest

    On 9/10/2010 9:16 PM, Ben Morrow wrote:
    >
    > Quoth :
    >> Ben Morrow wrote:
    >>>
    >>> There isn't any way to get open.pm to push any layer other than
    >>> :encoding or :utf8 onto the STD handles (no, I don't understand why
    >>> either). I think the best you're going to get is
    >>>
    >>> perl -lne'BEGIN{binmode$_ for<STD{IN,OUT}>;...}...'
    >>>
    >>> (now *there's* a nasty golf trick...).

    >>
    >> Yes but, Perl doesn't use STDIN with the -n or -p switch, it uses ARGV,
    >> and if you use the -i switch as well it uses ARGVOUT instead of STDOUT.

    >
    > True, and, good point :). It's not quite that simple, though:
    >
    > ~% export PERLIO=:perlio:crlf
    > ~% echo foo>foo
    > ~% perl -Mopen=IO,:raw -nE'say for PerlIO::get_layers \*ARGV' foo
    > unix
    > ~% echo foo | perl -Mopen=IO,:raw
    > -nE'say for PerlIO::get_layers \*ARGV'
    > unix
    > perlio
    > crlf
    > ~%
    >
    > and STDOUT will still need explicitly binmoding in cases where you're
    > using it. (I have no idea where the :perlio went in the first case.)


    conclusion: IO golfing is _hard_.

    --

    "Six by nine. Forty two."
    "That's it. That's all there is."
    "I always thought something was fundamentally wrong with the universe."
     
    sreservoir, Sep 11, 2010
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shane Wealti

    convert byte[] to Byte[]

    Shane Wealti, Jun 13, 2005, in forum: Java
    Replies:
    5
    Views:
    42,260
    Brzezi
    Jun 13, 2005
  2. Replies:
    20
    Views:
    9,812
    licebmi
    Sep 7, 2009
  3. Deep
    Replies:
    6
    Views:
    497
    Nick Keighley
    Feb 28, 2007
  4. dolphin
    Replies:
    6
    Views:
    561
    Thomas Fritsch
    Mar 18, 2007
  5. Roedy Green

    byte stream vs char stream buffer

    Roedy Green, May 7, 2014, in forum: Java
    Replies:
    20
    Views:
    206
    Silvio
    May 18, 2014
Loading...

Share This Page