Question about reading a big binary file and write it into several text (ascii) files

Discussion in 'Python' started by Albert Tu, Jan 24, 2005.

  1. Albert Tu

    Albert Tu Guest

    Hi,

    I am learning and pretty new to Python and I hope your guys can give me
    a quick start.

    I have an about 1G-byte binary file from a flat panel x-ray detector; I
    know at the beggining there is a 128-byte header and the rest of the
    file is integers in 2-byte format.

    What I want to do is to save the binary data into several smaller files
    in integer format and each smaller file has the size of 2*1024*768
    bytes.

    I know I can do something like
    >>>f=open("xray.seq", 'rb')
    >>>header=f.read(128)
    >>>file1=f.read(2*1024*768)
    >>>file2=f.read(2*1024*768)
    >>>......
    >>>f.close()


    Bur I don't them how to save files in integer format (converting from
    binary to ascii files) and how to do this in an elegant and snappy way.


    Please reply when you guyes can get a chance.
    Thanks,
    Warm regards,
    Albert
     
    Albert Tu, Jan 24, 2005
    #1
    1. Advertising

  2. Re: Question about reading a big binary file and write it intoseveraltext (ascii) files

    Albert Tu wrote:

    > I am learning and pretty new to Python and I hope your guys can give me
    > a quick start.
    >
    > I have an about 1G-byte binary file from a flat panel x-ray detector; I
    > know at the beggining there is a 128-byte header and the rest of the
    > file is integers in 2-byte format.
    >
    > What I want to do is to save the binary data into several smaller files
    > in integer format and each smaller file has the size of 2*1024*768
    > bytes.
    >
    > I know I can do something like
    >>>>f=open("xray.seq", 'rb')
    >>>>header=f.read(128)
    >>>>file1=f.read(2*1024*768)
    >>>>file2=f.read(2*1024*768)
    >>>>......


    (using a loop might help)

    >>>>f.close()

    >
    > Bur I don't them how to save files in integer format (converting from
    > binary to ascii files) and how to do this in an elegant and snappy way.


    I think you have to define "integer format" a bit better. A text file with
    integer values, written out in decimal?

    If so, take a look at the array module:

    http://docs.python.org/lib/module-array.html

    Here's an (untested) example; tweak as necessary:

    linesize = 1024
    data = array("h", filedata)
    for i in range(0, len(data), linesize):
    # convert to list of decimal integers
    list = map(str, data[i:i+linesize])
    print " ".join(list)

    tools like PIL and NumPy may also come in handy, but I suspect they're
    overkill in this case.

    </F>
     
    Fredrik Lundh, Jan 24, 2005
    #2
    1. Advertising

  3. On 24 Jan 2005 12:44:32 -0800, "Albert Tu" <> wrote:

    >Hi,
    >
    >I am learning and pretty new to Python and I hope your guys can give me
    >a quick start.
    >
    >I have an about 1G-byte binary file from a flat panel x-ray detector; I
    >know at the beggining there is a 128-byte header and the rest of the
    >file is integers in 2-byte format.

    It looks like 16-bit pixels in the 1024*768 images, I assume
    >
    >What I want to do is to save the binary data into several smaller files
    >in integer format and each smaller file has the size of 2*1024*768
    >bytes.

    You could do that, but why duplicate so much data that you may never look at?
    E.g., why not a class that provides a view of your big file in terms of an image index
    and returns an efficient array in memory e.g., (untested)

    import array
    def getimage(n, f, offset=128):
    f.seek(offset+n*2*1024*768)
    return array('H', f.read(2*1024*768)) # 'H' is for unsigned 2-byte integers (check endianness for swap need!)

    Then usage would be
    imfile = open('big_file.bin', 'rb')
    imarray = getimage(23, imfile)
    And you could get pixel x,y by
    xpix, ypix = imarray[x+y*1024] # or maybe x*768+y etc.

    or your could make getimage a method of a class that you intialize with
    the file and which could maintain an lru cache of images
    with a particular disk directory as backup, etc. etc. and would provide
    images wrapped with nice methods to support whatever you are doing with the images.


    >
    >I know I can do something like
    >>>>f=open("xray.seq", 'rb')
    >>>>header=f.read(128)
    >>>>file1=f.read(2*1024*768)
    >>>>file2=f.read(2*1024*768)
    >>>>......
    >>>>f.close()

    >
    >Bur I don't them how to save files in integer format (converting from
    >binary to ascii files) and how to do this in an elegant and snappy way.

    Best is probably to leave the original format alone, e.g., (untested and needs try/except)
    this should split the big file into individual image files named file0.ximg .. filen.ximg

    f = open('xray.seq/, 'rb')
    header = f.read(128)
    nfile = 0
    while 1:
    im = f.read(2*1024*768)
    if not im: break
    if len(im) != 2*1024*768: print 'broken tail of %s bytes'%len(im); break
    fw = open('file%s.ximg' % nfile, 'wb')
    fw.write(im)
    fw.close()
    nfile +=1

    then you could use getimage above with offset passed as 0 and image number 0, e.g.,

    im23 = getimage(0, open('file23.ximg','rb'), 0) # img 0, offset 0

    But then you might wonder about all those separate files, unless you want to
    put them on a series of CDs where they wouldn't all fit on one. Whatever ;-)

    You will probably lose in both speed and space if you try to make some kind
    of ascii disk files. You aren't thinking XML are you??!! For this, definitely ick ;-)

    >

    What you want to do will depend on the big picture, which is not apparent yet ;-)
    >
    >Please reply when you guyes can get a chance.
    >Thanks,


    Sorry to give nothing but untested suggestion, but I have to go, and I
    will be off line mostly for a while.

    Regards,
    Bengt Richter
     
    Bengt Richter, Jan 25, 2005
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. idiotprogrammer
    Replies:
    4
    Views:
    1,141
    Joseph Kesselman
    Mar 5, 2007
  2. Shaguf
    Replies:
    0
    Views:
    361
    Shaguf
    Dec 24, 2008
  3. Shaguf
    Replies:
    0
    Views:
    457
    Shaguf
    Dec 26, 2008
  4. someone
    Replies:
    37
    Views:
    2,521
    Joshua Maurice
    Oct 18, 2011
  5. Shaguf
    Replies:
    0
    Views:
    242
    Shaguf
    Dec 26, 2008
Loading...

Share This Page