Reading and writing a binary file

Discussion in 'Ruby' started by Chris Guenther, Mar 21, 2005.

  1. Hi,

    I am having trouble reading values from an existing (or maybe not yet
    exisiting) binary file:
    My target is to open a binary file for reading and writing (if it does not
    exisit it should be created).
    If the file was not yes existing I'd like to fill it with binary data to a
    certain predefined size.
    Later on I'd like to position my file position, and read and write whatever
    I like.

    The following piece does not work as intended:

    require 'pp'
    UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
    filename='foo.bin'
    size=1024
    dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
    begin
    fd=File.open(filename,"ab+")
    fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
    rescue => e
    pp e
    fd=File.open(filename,"wb+")
    puts "opened #{filename} with 'wib+'"
    end

    fd.sync=true
    fd.seek(0)
    unless dont_write
    bin_data = [0].pack('c')*(size)
    fd.write(bin_data)
    # OK now I really have a file of size 1024 with a binay 0 contents
    end

    ###.... then sometime later using the same file descriptor (fd) :
    requestes_stream_position = 900
    fd.pos=(requestes_stream_position)
    p "tell=#{fd.tell}"
    # Now please read 30 binary 0s into a binary string
    bin_value=fd.read(30)

    #### I ASSUMED THAT bin_value now is a "binary string of size 30", but
    unfortunately it is nil !!!



    Thanks in adcance,
    C.
     
    Chris Guenther, Mar 21, 2005
    #1
    1. Advertising

  2. "Chris Guenther" <> schrieb im Newsbeitrag
    news:d1mhu3$bi6$...
    > Hi,
    >
    > I am having trouble reading values from an existing (or maybe not yet
    > exisiting) binary file:
    > My target is to open a binary file for reading and writing (if it does

    not
    > exisit it should be created).
    > If the file was not yes existing I'd like to fill it with binary data to

    a
    > certain predefined size.
    > Later on I'd like to position my file position, and read and write

    whatever
    > I like.
    >
    > The following piece does not work as intended:
    >
    > require 'pp'
    > UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
    > filename='foo.bin'
    > size=1024
    > dont_write=FileTest.exists?(filename) &&

    File.stat(filename).size==size
    > begin
    > fd=File.open(filename,"ab+")
    > fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
    > rescue => e
    > pp e
    > fd=File.open(filename,"wb+")
    > puts "opened #{filename} with 'wib+'"
    > end
    >
    > fd.sync=true
    > fd.seek(0)
    > unless dont_write
    > bin_data = [0].pack('c')*(size)
    > fd.write(bin_data)
    > # OK now I really have a file of size 1024 with a binay 0 contents
    > end
    >
    > ###.... then sometime later using the same file descriptor (fd) :
    > requestes_stream_position = 900
    > fd.pos=(requestes_stream_position)
    > p "tell=#{fd.tell}"
    > # Now please read 30 binary 0s into a binary string
    > bin_value=fd.read(30)
    >
    > #### I ASSUMED THAT bin_value now is a "binary string of size 30",

    but
    > unfortunately it is nil !!!
    >
    >
    >
    > Thanks in adcance,
    > C.


    I don't know what you're doing wrong. Your script works fine for me.
    Btw, you don't close the fd.

    Here's how I'd do it:

    filename='foo.bin'
    size=1024

    File.open(filename, "ab+") do |fd|
    fd.seek(0, IO::SEEK_END)

    if fd.tell < size
    fd.seek 0
    fd.write( "\000" * size );
    elsif fd.tell > size
    # if you need the file to be exact this size
    fd.truncate size
    end

    requestes_stream_position = 900
    fd.seek requestes_stream_position

    bin_value = fd.read(30)
    p bin_value
    end

    Notes:

    - don't change binmode depending on platform, always use "b" for binary
    files.

    - use File.open with block for proper close / cleanup

    Kind regards

    robert
     
    Robert Klemme, Mar 21, 2005
    #2
    1. Advertising

  3. Thanks Robert,

    you are absolutely right with your remarks using a block. The code I
    posted was just a small copied fraction of the real code, in which I
    cannor use a block because the same fd will be stored in a hash for
    later usage by another method.
    Anyway, meanwhile I found the cause of this problem (again you are
    right, it was not in the code I posted) , but now I have another related
    problem that tricks my mind. I am trying to patch binary data in a file.
    Unfortunately the data I am patching is not the data I get when reading
    it back from the patched binary file.
    Any ideas??



    Thanks again in advance,
    Chris





    #!/usr/bin/env ruby

    FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    [?c,?a,?f,?e].pack('cccc')

    File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    bin_data = [0].pack('c')*FSIZE
    fd.write(bin_data)
    }

    File.truncate(FNAME,FSIZE)

    File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    data=fd.read(FSIZE)
    data[OFFSET,VALUE.length]=VALUE
    fd.pos=0
    fd.write(data)
    p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
    }

    File.truncate(FNAME,FSIZE)
    File.open(FNAME,"ab+") {|fd|
    fd.pos=OFFSET
    patched=fd.read(VALUE.length)
    p "read modified piece of data = '#{patched}'"
    puts "Oooops !!!" unless VALUE==patched
    }
     
    Chris Günther, Mar 22, 2005
    #3
  4. Chris Guenther wrote:
    > Hi,
    >
    > I am having trouble reading values from an existing (or maybe not yet
    > exisiting) binary file:
    > My target is to open a binary file for reading and writing (if it does not
    > exisit it should be created).
    > If the file was not yes existing I'd like to fill it with binary data to a
    > certain predefined size.
    > Later on I'd like to position my file position, and read and write whatever
    > I like.
    >
    > The following piece does not work as intended:
    >
    > require 'pp'
    > UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
    > filename='foo.bin'
    > size=1024
    > dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
    > begin
    > fd=File.open(filename,"ab+")
    > fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
    > rescue => e
    > pp e
    > fd=File.open(filename,"wb+")
    > puts "opened #{filename} with 'wib+'"
    > end
    >
    > fd.sync=true
    > fd.seek(0)
    > unless dont_write
    > bin_data = [0].pack('c')*(size)
    > fd.write(bin_data)
    > # OK now I really have a file of size 1024 with a binay 0 contents
    > end
    >
    > ###.... then sometime later using the same file descriptor (fd) :
    > requestes_stream_position = 900
    > fd.pos=(requestes_stream_position)
    > p "tell=#{fd.tell}"
    > # Now please read 30 binary 0s into a binary string
    > bin_value=fd.read(30)
    >
    > #### I ASSUMED THAT bin_value now is a "binary string of size 30", but
    > unfortunately it is nil !!!
    >
    >
    >
    > Thanks in adcance,
    > C.
    >
    >
    >
    >
    >

    Thanks Robert,

    you are absolutely right with your remarks using a block. The code I
    posted was just a small copied fraction of the real code, in which I
    cannor use a block because the same fd will be stored in a hash for
    later usage by another method.
    Anyway, meanwhile I found the cause of this problem (again you are
    right, it was not in the code I posted) , but now I have another related
    problem that tricks my mind. I am trying to patch binary data in a file.
    Unfortunately the data I am patching is not the data I get when reading
    it back from the patched binary file.
    Any ideas??



    Thanks again in advance,
    Chris





    #!/usr/bin/env ruby

    FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    [?c,?a,?f,?e].pack('cccc')

    File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    bin_data = [0].pack('c')*FSIZE
    fd.write(bin_data)
    }

    File.truncate(FNAME,FSIZE)

    File.open(FNAME,"ab+") {|fd|
    fd.pos=0
    data=fd.read(FSIZE)
    data[OFFSET,VALUE.length]=VALUE
    fd.pos=0
    fd.write(data)
    p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
    }

    File.truncate(FNAME,FSIZE)
    File.open(FNAME,"ab+") {|fd|
    fd.pos=OFFSET
    patched=fd.read(VALUE.length)
    p "read modified piece of data = '#{patched}'"
    puts "Oooops !!!" unless VALUE==patched
    }
     
    Chris Günther, Mar 22, 2005
    #4
  5. "Chris Günther" <> schrieb im Newsbeitrag
    news:...
    > Thanks Robert,
    >
    > you are absolutely right with your remarks using a block. The code I
    > posted was just a small copied fraction of the real code, in which I
    > cannor use a block because the same fd will be stored in a hash for
    > later usage by another method.
    > Anyway, meanwhile I found the cause of this problem (again you are
    > right, it was not in the code I posted) , but now I have another related
    > problem that tricks my mind. I am trying to patch binary data in a file.
    > Unfortunately the data I am patching is not the data I get when reading
    > it back from the patched binary file.
    > Any ideas??
    >
    >
    >
    > Thanks again in advance,
    > Chris
    >
    >
    >
    >
    >
    > #!/usr/bin/env ruby
    >
    > FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    > [?c,?a,?f,?e].pack('cccc')
    >
    > File.open(FNAME,"ab+") {|fd|
    > fd.pos=0
    > bin_data = [0].pack('c')*FSIZE
    > fd.write(bin_data)
    > }
    >
    > File.truncate(FNAME,FSIZE)
    >
    > File.open(FNAME,"ab+") {|fd|
    > fd.pos=0
    > data=fd.read(FSIZE)
    > data[OFFSET,VALUE.length]=VALUE
    > fd.pos=0
    > fd.write(data)
    > p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
    > }


    You truncate too often:
    > File.truncate(FNAME,FSIZE)


    > File.open(FNAME,"ab+") {|fd|
    > fd.pos=OFFSET
    > patched=fd.read(VALUE.length)
    > p "read modified piece of data = '#{patched}'"
    > puts "Oooops !!!" unless VALUE==patched
    > }


    Careful when you're copying and pasting.

    Regards

    robert
     
    Robert Klemme, Mar 22, 2005
    #5
  6. Robert Klemme wrote:
    > "Chris Günther" <> schrieb im Newsbeitrag
    > news:...
    >
    >>Thanks Robert,
    >>
    >>you are absolutely right with your remarks using a block. The code I
    >>posted was just a small copied fraction of the real code, in which I
    >>cannor use a block because the same fd will be stored in a hash for
    >>later usage by another method.
    >>Anyway, meanwhile I found the cause of this problem (again you are
    >>right, it was not in the code I posted) , but now I have another related
    >>problem that tricks my mind. I am trying to patch binary data in a file.
    >>Unfortunately the data I am patching is not the data I get when reading
    >>it back from the patched binary file.
    >>Any ideas??
    >>
    >>
    >>
    >>Thanks again in advance,
    >>Chris
    >>
    >>
    >>
    >>
    >>
    >>#!/usr/bin/env ruby
    >>
    >>FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    >>[?c,?a,?f,?e].pack('cccc')
    >>
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=0
    >> bin_data = [0].pack('c')*FSIZE
    >> fd.write(bin_data)
    >>}
    >>
    >>File.truncate(FNAME,FSIZE)
    >>
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=0
    >> data=fd.read(FSIZE)
    >> data[OFFSET,VALUE.length]=VALUE
    >> fd.pos=0
    >> fd.write(data)
    >> p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
    >>}

    >
    >
    > You truncate too often:
    >
    >>File.truncate(FNAME,FSIZE)

    >
    >
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=OFFSET
    >> patched=fd.read(VALUE.length)
    >> p "read modified piece of data = '#{patched}'"
    >> puts "Oooops !!!" unless VALUE==patched
    >>}

    >
    >
    > Careful when you're copying and pasting.
    >
    > Regards
    >
    > robert
    >


    I dont see your point. I have to truncate after every write to ensure
    the file size does not change. Where is the connection to the problem
    that the read value does not match the written value ?

    regards,
    C.
     
    Chris Günther, Mar 22, 2005
    #6
  7. Robert Klemme wrote:
    > "Chris Günther" <> schrieb im Newsbeitrag
    > news:...
    >
    >>Thanks Robert,
    >>
    >>you are absolutely right with your remarks using a block. The code I
    >>posted was just a small copied fraction of the real code, in which I
    >>cannor use a block because the same fd will be stored in a hash for
    >>later usage by another method.
    >>Anyway, meanwhile I found the cause of this problem (again you are
    >>right, it was not in the code I posted) , but now I have another related
    >>problem that tricks my mind. I am trying to patch binary data in a file.
    >>Unfortunately the data I am patching is not the data I get when reading
    >>it back from the patched binary file.
    >>Any ideas??
    >>
    >>
    >>
    >>Thanks again in advance,
    >>Chris
    >>
    >>
    >>
    >>
    >>
    >>#!/usr/bin/env ruby
    >>
    >>FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    >>[?c,?a,?f,?e].pack('cccc')
    >>
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=0
    >> bin_data = [0].pack('c')*FSIZE
    >> fd.write(bin_data)
    >>}
    >>
    >>File.truncate(FNAME,FSIZE)
    >>
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=0
    >> data=fd.read(FSIZE)
    >> data[OFFSET,VALUE.length]=VALUE
    >> fd.pos=0
    >> fd.write(data)
    >> p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
    >>}

    >
    >
    > You truncate too often:
    >
    >>File.truncate(FNAME,FSIZE)

    >
    >
    >>File.open(FNAME,"ab+") {|fd|
    >> fd.pos=OFFSET
    >> patched=fd.read(VALUE.length)
    >> p "read modified piece of data = '#{patched}'"
    >> puts "Oooops !!!" unless VALUE==patched
    >>}

    >
    >
    > Careful when you're copying and pasting.
    >
    > Regards
    >
    > robert
    >


    I dont see your point. I have to truncate after every write to ensure
    the file size does not change. Where is the connection to the problem
    that the read value does not match the written value ?

    regards,
    C.
     
    Chris Günther, Mar 22, 2005
    #7
  8. Chris Guenther

    Pit Capitain Guest

    Chris Günther schrieb:
    > ... I am trying to patch binary data in a file.
    > Unfortunately the data I am patching is not the data I get when reading
    > it back from the patched binary file.
    > Any ideas??
    > ...
    > File.open(FNAME,"ab+") {|fd|
    > fd.pos=0
    > ...


    Hi Chris,

    it seems that the mode "ab+" is your problem. According to [1], "a+"
    means open the file for reading and writing, starting at the end of the
    file. This also means (at least on windows), that the end of file is
    position 0. So what you are doing above is writing the patched data
    after the end of the original file.

    For patching a binary file I'd change your code to:

    FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    [?c,?a,?f,?e].pack('cccc')

    # "wb+" for initial data: truncate existing content
    File.open(FNAME,"wb+") {|fd|
    bin_data = [0].pack('c')*FSIZE
    fd.write(bin_data)
    }

    # "rb+" for the patch: open at the beginning
    File.open(FNAME,"rb+") {|fd|
    fd.pos=OFFSET
    fd.write(VALUE)
    p "written modified piece of data = '#{VALUE}'"
    }

    # "rb" for reading the data
    File.open(FNAME,"rb") {|fd|
    fd.pos=OFFSET
    patched=fd.read(VALUE.length)
    p "read modified piece of data = '#{patched}'"
    puts "Oooops !!!" unless VALUE==patched
    }

    Regards,
    Pit

    [1] http://www.ruby-doc.org/core/classes/IO.html
     
    Pit Capitain, Mar 22, 2005
    #8
  9. Chris Guenther

    Carlos Guest

    [Chris Günther <>, 2005-03-22 12.29 CET]
    > Thanks Robert,
    >
    > you are absolutely right with your remarks using a block. The code I
    > posted was just a small copied fraction of the real code, in which I
    > cannor use a block because the same fd will be stored in a hash for
    > later usage by another method.
    > Anyway, meanwhile I found the cause of this problem (again you are
    > right, it was not in the code I posted) , but now I have another related
    > problem that tricks my mind. I am trying to patch binary data in a file.
    > Unfortunately the data I am patching is not the data I get when reading
    > it back from the patched binary file.
    > Any ideas??


    All your problems come from the fact that your are opening the file for
    appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
    if you want, but it will always write at the end of file.
     
    Carlos, Mar 22, 2005
    #9
  10. "Chris Günther" <> schrieb im Newsbeitrag
    news:...

    <snip/>

    > I dont see your point. I have to truncate after every write to ensure
    > the file size does not change. Where is the connection to the problem
    > that the read value does not match the written value ?


    I don't see the point either. :-} Sorry for the confusion, I overlooked
    that you used the other form of truncate with a length.

    Regards

    robert
     
    Robert Klemme, Mar 23, 2005
    #10
  11. Pit Capitain wrote:
    > Chris Günther schrieb:
    >
    >> ... I am trying to patch binary data in a file.
    >> Unfortunately the data I am patching is not the data I get when
    >> reading it back from the patched binary file.
    >> Any ideas??
    >> ...
    >> File.open(FNAME,"ab+") {|fd|
    >> fd.pos=0
    >> ...

    >
    >
    > Hi Chris,
    >
    > it seems that the mode "ab+" is your problem. According to [1], "a+"
    > means open the file for reading and writing, starting at the end of the
    > file. This also means (at least on windows), that the end of file is
    > position 0. So what you are doing above is writing the patched data
    > after the end of the original file.
    >
    > For patching a binary file I'd change your code to:
    >
    > FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
    > [?c,?a,?f,?e].pack('cccc')
    >
    > # "wb+" for initial data: truncate existing content
    > File.open(FNAME,"wb+") {|fd|
    > bin_data = [0].pack('c')*FSIZE
    > fd.write(bin_data)
    > }
    >
    > # "rb+" for the patch: open at the beginning
    > File.open(FNAME,"rb+") {|fd|
    > fd.pos=OFFSET
    > fd.write(VALUE)
    > p "written modified piece of data = '#{VALUE}'"
    > }
    >
    > # "rb" for reading the data
    > File.open(FNAME,"rb") {|fd|
    > fd.pos=OFFSET
    > patched=fd.read(VALUE.length)
    > p "read modified piece of data = '#{patched}'"
    > puts "Oooops !!!" unless VALUE==patched
    > }
    >
    > Regards,
    > Pit
    >
    > [1] http://www.ruby-doc.org/core/classes/IO.html
    >
    >

    Hi Pit,
    thanks for your contribution.
    That's exactly the solution I already have. What I dont like about this
    solution is that you have to open and close file descriptors 3 times
    just for patching one little file (For sure a working solution but quite
    some overhead). Just for academical reason it seeems I have to fall back
    reading the C-sources :-( to find out what is going wrong under the hood.

    Chris
     
    Chris Günther, Mar 23, 2005
    #11
  12. Carlos wrote:
    > [Chris Günther <>, 2005-03-22 12.29 CET]
    >
    >>Thanks Robert,
    >>
    >>you are absolutely right with your remarks using a block. The code I
    >>posted was just a small copied fraction of the real code, in which I
    >>cannor use a block because the same fd will be stored in a hash for
    >>later usage by another method.
    >>Anyway, meanwhile I found the cause of this problem (again you are
    >>right, it was not in the code I posted) , but now I have another related
    >>problem that tricks my mind. I am trying to patch binary data in a file.
    >>Unfortunately the data I am patching is not the data I get when reading
    >>it back from the patched binary file.
    >>Any ideas??

    >
    >
    > All your problems come from the fact that your are opening the file for
    > appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
    > if you want, but it will always write at the end of file.
    >
    >
    >

    Ok, I got it - (maybe only for me) this kind of semanctics is really
    confusing (and not well described either).

    Anyway thanks a lot,
    Chris
     
    Chris Günther, Mar 23, 2005
    #12
  13. Chris Guenther

    Pit Capitain Guest

    Chris Günther schrieb:
    > That's exactly the solution I already have. What I dont like about this
    > solution is that you have to open and close file descriptors 3 times
    > just for patching one little file (For sure a working solution but quite
    > some overhead).


    In order to patch an existing file, all you have to do is

    >> File.open(FNAME,"rb+") {|fd|
    >> fd.pos=OFFSET
    >> fd.write(VALUE)
    >> p "written modified piece of data = '#{VALUE}'"
    >> }


    The other File.open calls are for creating the test file and for testing
    the contents of the patched file. I copied them from your initial
    script. Of course you could do all three steps (create, patch, test) in
    a single File.open block.

    Regards,
    Pit
     
    Pit Capitain, Mar 23, 2005
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Daniel Gowans

    Reading/Writing pure binary files

    Daniel Gowans, May 27, 2004, in forum: VHDL
    Replies:
    2
    Views:
    5,181
  2. Steve Bennett

    need help reading/writing binary

    Steve Bennett, Aug 25, 2004, in forum: Perl
    Replies:
    1
    Views:
    498
    Joe Smith
    Aug 25, 2004
  3. Louis
    Replies:
    6
    Views:
    1,093
    Sudsy
    Oct 15, 2003
  4. Daniel Moree

    Reading and Writing to Binary Files

    Daniel Moree, Nov 23, 2004, in forum: C++
    Replies:
    9
    Views:
    1,007
    Jonathan Mcdougall
    Nov 24, 2004
  5. Ron Eggler

    writing binary file (ios::binary)

    Ron Eggler, Apr 25, 2008, in forum: C++
    Replies:
    9
    Views:
    935
    James Kanze
    Apr 28, 2008
Loading...

Share This Page