Reading and writing a binary file

[Chris Guenther]

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does not
exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to a
certain predefined size.
Later on I'd like to position my file position, and read and write whatever
I like.

The following piece does not work as intended:

require 'pp'
UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
filename='foo.bin'
size=1024
dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
begin
fd=File.open(filename,"ab+")
fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
rescue => e
pp e
fd=File.open(filename,"wb+")
puts "opened #{filename} with 'wib+'"
end

fd.sync=true
fd.seek(0)
unless dont_write
bin_data = [0].pack('c')*(size)
fd.write(bin_data)
# OK now I really have a file of size 1024 with a binay 0 contents
end

###.... then sometime later using the same file descriptor (fd) :
requestes_stream_position = 900
fd.pos=(requestes_stream_position)
p "tell=#{fd.tell}"
# Now please read 30 binary 0s into a binary string
bin_value=fd.read(30)

#### I ASSUMED THAT bin_value now is a "binary string of size 30", but
unfortunately it is nil !!!



Thanks in adcance,
C.

 

[Robert Klemme]

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does not
exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to a
certain predefined size.
Later on I'd like to position my file position, and read and write whatever
I like.

The following piece does not work as intended:

require 'pp'
UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
filename='foo.bin'
size=1024
dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
begin
fd=File.open(filename,"ab+")
fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
rescue => e
pp e
fd=File.open(filename,"wb+")
puts "opened #{filename} with 'wib+'"
end

fd.sync=true
fd.seek(0)
unless dont_write
bin_data = [0].pack('c')*(size)
fd.write(bin_data)
# OK now I really have a file of size 1024 with a binay 0 contents
end

###.... then sometime later using the same file descriptor (fd) :
requestes_stream_position = 900
fd.pos=(requestes_stream_position)
p "tell=#{fd.tell}"
# Now please read 30 binary 0s into a binary string
bin_value=fd.read(30)

#### I ASSUMED THAT bin_value now is a "binary string of size 30", but
unfortunately it is nil !!!



Thanks in adcance,
C.

I don't know what you're doing wrong. Your script works fine for me.
Btw, you don't close the fd.

Here's how I'd do it:

filename='foo.bin'
size=1024

File.open(filename, "ab+") do |fd|
fd.seek(0, IO::SEEK_END)

if fd.tell < size
fd.seek 0
fd.write( "\000" * size );
elsif fd.tell > size
# if you need the file to be exact this size
fd.truncate size
end

requestes_stream_position = 900
fd.seek requestes_stream_position

bin_value = fd.read(30)
p bin_value
end

Notes:

- don't change binmode depending on platform, always use "b" for binary
files.

- use File.open with block for proper close / cleanup

Kind regards

robert

 

[Chris Günther]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??



Thanks again in advance,
Chris





#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
fd.pos=0
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=0
data=fd.read(FSIZE)
data[OFFSET,VALUE.length]=VALUE
fd.pos=0
fd.write(data)
p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

File.truncate(FNAME,FSIZE)
File.open(FNAME,"ab+") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

 

[Chris Günther]

Hi,

I am having trouble reading values from an existing (or maybe not yet
exisiting) binary file:
My target is to open a binary file for reading and writing (if it does not
exisit it should be created).
If the file was not yes existing I'd like to fill it with binary data to a
certain predefined size.
Later on I'd like to position my file position, and read and write whatever
I like.

The following piece does not work as intended:

require 'pp'
UNDER_UNIX = CONFIG['target_os'].to_s.index('linux')!=nil
filename='foo.bin'
size=1024
dont_write=FileTest.exists?(filename) && File.stat(filename).size==size
begin
fd=File.open(filename,"ab+")
fd.binmode unless UNDER_UNIX # assuming !UNIX=WINDOWS
rescue => e
pp e
fd=File.open(filename,"wb+")
puts "opened #{filename} with 'wib+'"
end

fd.sync=true
fd.seek(0)
unless dont_write
bin_data = [0].pack('c')*(size)
fd.write(bin_data)
# OK now I really have a file of size 1024 with a binay 0 contents
end

###.... then sometime later using the same file descriptor (fd) :
requestes_stream_position = 900
fd.pos=(requestes_stream_position)
p "tell=#{fd.tell}"
# Now please read 30 binary 0s into a binary string
bin_value=fd.read(30)

#### I ASSUMED THAT bin_value now is a "binary string of size 30", but
unfortunately it is nil !!!



Thanks in adcance,
C.

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??



Thanks again in advance,
Chris





#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
fd.pos=0
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=0
data=fd.read(FSIZE)
data[OFFSET,VALUE.length]=VALUE
fd.pos=0
fd.write(data)
p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

File.truncate(FNAME,FSIZE)
File.open(FNAME,"ab+") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

 

[Robert Klemme]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??



Thanks again in advance,
Chris





#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
fd.pos=0
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=0
data=fd.read(FSIZE)
data[OFFSET,VALUE.length]=VALUE
fd.pos=0
fd.write(data)
p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

robert

 

[Chris Günther]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??



Thanks again in advance,
Chris





#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
fd.pos=0
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=0
data=fd.read(FSIZE)
data[OFFSET,VALUE.length]=VALUE
fd.pos=0
fd.write(data)
p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

robert

I dont see your point. I have to truncate after every write to ensure
the file size does not change. Where is the connection to the problem
that the read value does not match the written value ?

regards,
C.

 

[Chris Günther]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??



Thanks again in advance,
Chris





#!/usr/bin/env ruby

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

File.open(FNAME,"ab+") {|fd|
fd.pos=0
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=0
data=fd.read(FSIZE)
data[OFFSET,VALUE.length]=VALUE
fd.pos=0
fd.write(data)
p "written modified piece of data = '#{data[OFFSET,VALUE.length]}'"
}

You truncate too often:

File.truncate(FNAME,FSIZE)

File.open(FNAME,"ab+") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

Careful when you're copying and pasting.

Regards

robert

I dont see your point. I have to truncate after every write to ensure
the file size does not change. Where is the connection to the problem
that the read value does not match the written value ?

regards,
C.

 

[Pit Capitain]

... I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??
...
File.open(FNAME,"ab+") {|fd|
fd.pos=0
...

Hi Chris,

it seems that the mode "ab+" is your problem. According to [1], "a+"
means open the file for reading and writing, starting at the end of the
file. This also means (at least on windows), that the end of file is
position 0. So what you are doing above is writing the patched data
after the end of the original file.

For patching a binary file I'd change your code to:

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

# "wb+" for initial data: truncate existing content
File.open(FNAME,"wb+") {|fd|
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

# "rb+" for the patch: open at the beginning
File.open(FNAME,"rb+") {|fd|
fd.pos=OFFSET
fd.write(VALUE)
p "written modified piece of data = '#{VALUE}'"
}

# "rb" for reading the data
File.open(FNAME,"rb") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

Regards,
Pit

[1] http://www.ruby-doc.org/core/classes/IO.html

 

[Carlos]

Thanks Robert,

you are absolutely right with your remarks using a block. The code I
posted was just a small copied fraction of the real code, in which I
cannor use a block because the same fd will be stored in a hash for
later usage by another method.
Anyway, meanwhile I found the cause of this problem (again you are
right, it was not in the code I posted) , but now I have another related
problem that tricks my mind. I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when reading
it back from the patched binary file.
Any ideas??

All your problems come from the fact that your are opening the file for
appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
if you want, but it will always write at the end of file.

 

[Robert Klemme]

I dont see your point. I have to truncate after every write to ensure
the file size does not change. Where is the connection to the problem
that the read value does not match the written value ?

I don't see the point either. :-} Sorry for the confusion, I overlooked
that you used the other form of truncate with a length.

Regards

robert

 

[Chris Günther]

... I am trying to patch binary data in a file.
Unfortunately the data I am patching is not the data I get when
reading it back from the patched binary file.
Any ideas??
...
File.open(FNAME,"ab+") {|fd|
fd.pos=0
...

Hi Chris,

it seems that the mode "ab+" is your problem. According to [1], "a+"
means open the file for reading and writing, starting at the end of the
file. This also means (at least on windows), that the end of file is
position 0. So what you are doing above is writing the patched data
after the end of the original file.

For patching a binary file I'd change your code to:

FNAME, FSIZE, OFFSET, VALUE = 'foooo.dat', 1024, 32,
[?c,?a,?f,?e].pack('cccc')

# "wb+" for initial data: truncate existing content
File.open(FNAME,"wb+") {|fd|
bin_data = [0].pack('c')*FSIZE
fd.write(bin_data)
}

# "rb+" for the patch: open at the beginning
File.open(FNAME,"rb+") {|fd|
fd.pos=OFFSET
fd.write(VALUE)
p "written modified piece of data = '#{VALUE}'"
}

# "rb" for reading the data
File.open(FNAME,"rb") {|fd|
fd.pos=OFFSET
patched=fd.read(VALUE.length)
p "read modified piece of data = '#{patched}'"
puts "Oooops !!!" unless VALUE==patched
}

Regards,
Pit

[1] http://www.ruby-doc.org/core/classes/IO.html

Hi Pit,
thanks for your contribution.
That's exactly the solution I already have. What I dont like about this
solution is that you have to open and close file descriptors 3 times
just for patching one little file (For sure a working solution but quite
some overhead). Just for academical reason it seeems I have to fall back
reading the C-sources :-( to find out what is going wrong under the hood.

Chris

 

[Chris Günther]

All your problems come from the fact that your are opening the file for
appending ("ab+") instead of for reading ("r+b"). With "ab+", you can set fd.pos=0,
if you want, but it will always write at the end of file.

Ok, I got it - (maybe only for me) this kind of semanctics is really
confusing (and not well described either).

Anyway thanks a lot,
Chris

 

[Pit Capitain]

That's exactly the solution I already have. What I dont like about this
solution is that you have to open and close file descriptors 3 times
just for patching one little file (For sure a working solution but quite
some overhead).

In order to patch an existing file, all you have to do is

The other File.open calls are for creating the test file and for testing
the contents of the patched file. I copied them from your initial
script. Of course you could do all three steps (create, patch, test) in
a single File.open block.

Regards,
Pit