Noobie: Open file -> read characters & multiply

[gonzlobo]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely hosed!
decodedFile.write(time)

#close files
AP_File.close()
decoded_File.close()
---
AP.txt
000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
000000d5 26 0601 80 00 80 00 02 37 fe 54 01 09
000000d5 06 0602 80 00 e0 00 01 29 fe d2 69 99
000000d5 06 0603 80 00 e0 00 02 29 fe d2 6a 99
000000d5 26 0604 80 00 fe 54 02 09 80 00 01 5d
000000d5 06 0605 80 00 e0 00 02 15 fc 71 ca 0b
000000d5 4a 0610 81 00 86 00 02 26 12 00 02 a6
000000d5 4f 0611 00 00 00 50 00 00 00 00 07 00
000000d5 06 0612 80 00 e0 00 01 15 fc 71 c9 0b
000000d5 0a 0613 08 5c 04 88 08 98 00 00 00 00
000000d5 06 0614 80 00 e0 00 02 01 60 79 82 2b
000000d5 0a 0615 08 00 00 00 00 00 00 00 00 00
000000d5 26 0616 80 00 80 00 02 5d 04 22 3a 88
(actual files are 250MB!)

decodedTime.txt (should be)
0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
....

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

Any help is really appreciated.

 

[Steven D'Aprano]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely hosed!
decodedFile.write(time)

What does "this line is completely hosed!" mean? Does it crash your PC?
Does it raise an exception? Do the wrong thing?

Try this:

for line in AP_File:
# Typical line looks like this:
# 000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
# This should convert to:
# 0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
time = int(line[0:8], 16) * 0.0001
line = str(time) + line[8:]
decodedFile.write(line)

You might need to think about how many decimal places you want the time to
store.

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

You can convert this:

AP_File= file("AP.txt", "r") # file is recommended over "open"
line = AP_File.readline()
while line != "":
do_something_with_line
line = AP_File.readline()
AP_File.close()


into this:

AP_File= file("AP.txt", "r")
for line in AP_File:
do_something_with_line
AP_File.close()

You need to look at what your boss does with the lines, not how he does
the loop.

 

[Scott David Daniels]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely
hosed!
decodedFile.write(time)

#close files
AP_File.close()
decoded_File.close()
---
AP.txt
000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
000000d5 26 0601 80 00 80 00 02 37 fe 54 01 09
000000d5 06 0602 80 00 e0 00 01 29 fe d2 69 99
000000d5 06 0603 80 00 e0 00 02 29 fe d2 6a 99
000000d5 26 0604 80 00 fe 54 02 09 80 00 01 5d
000000d5 06 0605 80 00 e0 00 02 15 fc 71 ca 0b
000000d5 4a 0610 81 00 86 00 02 26 12 00 02 a6
000000d5 4f 0611 00 00 00 50 00 00 00 00 07 00
000000d5 06 0612 80 00 e0 00 01 15 fc 71 c9 0b
000000d5 0a 0613 08 5c 04 88 08 98 00 00 00 00
000000d5 06 0614 80 00 e0 00 02 01 60 79 82 2b
000000d5 0a 0615 08 00 00 00 00 00 00 00 00 00
000000d5 26 0616 80 00 80 00 02 5d 04 22 3a 88
(actual files are 250MB!)

decodedTime.txt (should be)
0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
...

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

Any help is really appreciated.

for line in AP_file:
print >>decoded_File, '%s.%04d' % divmod(int(line[:8], 16), 10000
), line[9:].rstrip()

or:

for line in AP_file:
print >>decoded_File, '%.4f' % (int(line[:8], 16) * .0001
), line[9:].rstrip()


--Scott David Daniels
(e-mail address removed)

 

[WaterWalk]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely hosed!
decodedFile.write(time)

#close files
AP_File.close()
decoded_File.close()
---
AP.txt
000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
000000d5 26 0601 80 00 80 00 02 37 fe 54 01 09
000000d5 06 0602 80 00 e0 00 01 29 fe d2 69 99
000000d5 06 0603 80 00 e0 00 02 29 fe d2 6a 99
000000d5 26 0604 80 00 fe 54 02 09 80 00 01 5d
000000d5 06 0605 80 00 e0 00 02 15 fc 71 ca 0b
000000d5 4a 0610 81 00 86 00 02 26 12 00 02 a6
000000d5 4f 0611 00 00 00 50 00 00 00 00 07 00
000000d5 06 0612 80 00 e0 00 01 15 fc 71 c9 0b
000000d5 0a 0613 08 5c 04 88 08 98 00 00 00 00
000000d5 06 0614 80 00 e0 00 02 01 60 79 82 2b
000000d5 0a 0615 08 00 00 00 00 00 00 00 00 00
000000d5 26 0616 80 00 80 00 02 5d 04 22 3a 88
(actual files are 250MB!)

decodedTime.txt (should be)
0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
...

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

Any help is really appreciated.

Use the built-in int(). It has an optional argument "radix" which
specifies the base for the conversion. For example:

 

[WaterWalk]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely hosed!
decodedFile.write(time)

#close files
AP_File.close()
decoded_File.close()
---
AP.txt
000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
000000d5 26 0601 80 00 80 00 02 37 fe 54 01 09
000000d5 06 0602 80 00 e0 00 01 29 fe d2 69 99
000000d5 06 0603 80 00 e0 00 02 29 fe d2 6a 99
000000d5 26 0604 80 00 fe 54 02 09 80 00 01 5d
000000d5 06 0605 80 00 e0 00 02 15 fc 71 ca 0b
000000d5 4a 0610 81 00 86 00 02 26 12 00 02 a6
000000d5 4f 0611 00 00 00 50 00 00 00 00 07 00
000000d5 06 0612 80 00 e0 00 01 15 fc 71 c9 0b
000000d5 0a 0613 08 5c 04 88 08 98 00 00 00 00
000000d5 06 0614 80 00 e0 00 02 01 60 79 82 2b
000000d5 0a 0615 08 00 00 00 00 00 00 00 00 00
000000d5 26 0616 80 00 80 00 02 5d 04 22 3a 88
(actual files are 250MB!)

decodedTime.txt (should be)
0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
...

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

Any help is really appreciated.

Use the built-in int(). It has an optional argument "radix" which
specifies the base for the conversion. For example:

Oh I forget that ">>>" will cause the line to be hidden by default. The
example is:
int("0x0A", 16) # will return 10

 

[WaterWalk]

I've been using Python for a few days. It's such the perfect language
for parsing data!

I really like it so far, but I'm having a hard time reading a file,
reading the first few hex characters & converting them to an integer.
Once the characters are converted to an integer, I'd like to write the
data to another file.

Here's the code snipped to the bare minimum:
---
# Open File
AP_File= open("AP.txt", "r")
decoded_File= open("decoded.txt", "w")

# read & process data line by line
for line in AP_File:
time = int(hex(line[0:8]), 16) * 0.0001 # this line is completely hosed!
decodedFile.write(time)

#close files
AP_File.close()
decoded_File.close()
---
AP.txt
000000d5 26 0600 80 00 ec 80 02 03 7d db 02 33
000000d5 26 0601 80 00 80 00 02 37 fe 54 01 09
000000d5 06 0602 80 00 e0 00 01 29 fe d2 69 99
000000d5 06 0603 80 00 e0 00 02 29 fe d2 6a 99
000000d5 26 0604 80 00 fe 54 02 09 80 00 01 5d
000000d5 06 0605 80 00 e0 00 02 15 fc 71 ca 0b
000000d5 4a 0610 81 00 86 00 02 26 12 00 02 a6
000000d5 4f 0611 00 00 00 50 00 00 00 00 07 00
000000d5 06 0612 80 00 e0 00 01 15 fc 71 c9 0b
000000d5 0a 0613 08 5c 04 88 08 98 00 00 00 00
000000d5 06 0614 80 00 e0 00 02 01 60 79 82 2b
000000d5 0a 0615 08 00 00 00 00 00 00 00 00 00
000000d5 26 0616 80 00 80 00 02 5d 04 22 3a 88
(actual files are 250MB!)

decodedTime.txt (should be)
0.0213 26 0600 80 00 ec 80 02 03 7d db 02 33
...

My boss and I are trying to complete the same task (he figured out how
to do it, but his code uses a while != "" loop and doesn't look
pythony (it looks too 'c'). Not that there's anything wrong with that!

Any help is really appreciated.

Use the built-in int(). It has an optional argument "radix" which
specifies the base for the conversion. For example:

int("0x0A", 16)
10

Oh I forget that ">>>" will cause the line to be hidden by default. The
example is:
int("0x0A", 16) # will return 10

I misunderstand the question, sorry for this.

Why not just split the line read since each number is separated by
space or tab. After splitting there is a list of numbers, then convert
the first element and write the list into a file.

 

[gonzlobo]

Thanks to all that responded. I chose a modified version of Scott's
second recommendation:

time = line[:8]
decoded_File.write( '%00.4f' % (int(time, 16) * .0001) + ', ')

'print >>' added a CRLF that I didn't need, so I went with '.print' (I
need to process about 20 values from the remaining bytes).

Thank you.

Any help is really appreciated.

for line in AP_file:
print >>decoded_File, '%s.%04d' % divmod(int(line[:8], 16), 10000
), line[9:].rstrip()

or:

for line in AP_file:
print >>decoded_File, '%.4f' % (int(line[:8], 16) * .0001
), line[9:].rstrip()

....