A
andrea crotti
We're having some really obscure problems with gzip.
There is a program running with python2.7 on a 2.6.18-128.el5xen (red
hat I think) kernel.
Now this program does the following:
if filename == 'out2.txt':
out2 = open('out2.txt')
elif filename == 'out2.txt.gz'
out2 = open('out2.txt.gz')
text = out2.read()
out2.close()
very simple right? But sometimes we get a checksum error.
Reading the code I got the following:
- CRC is at the end of the file and is computed against the whole
file (last 8 bytes)
- after the CRC there is the \0000 marker for the EOF
- readline() doesn't trigger the checksum generation in the
beginning, but only when the EOF is reached
- until a file is flushed or closed you can't read the new content in it
but the problem is that we can't reproduce it, because doing it
manually on the same files it works perfectly,
and the same files some time work some time don't work.
The files are on a shared NFS drive, I'm starting to think that it's a
network/fs problem, which might truncate the file
adding an EOF before the end and thus making the checksum fail..
But is it possible?
Or what else could it be?
There is a program running with python2.7 on a 2.6.18-128.el5xen (red
hat I think) kernel.
Now this program does the following:
if filename == 'out2.txt':
out2 = open('out2.txt')
elif filename == 'out2.txt.gz'
out2 = open('out2.txt.gz')
text = out2.read()
out2.close()
very simple right? But sometimes we get a checksum error.
Reading the code I got the following:
- CRC is at the end of the file and is computed against the whole
file (last 8 bytes)
- after the CRC there is the \0000 marker for the EOF
- readline() doesn't trigger the checksum generation in the
beginning, but only when the EOF is reached
- until a file is flushed or closed you can't read the new content in it
but the problem is that we can't reproduce it, because doing it
manually on the same files it works perfectly,
and the same files some time work some time don't work.
The files are on a shared NFS drive, I'm starting to think that it's a
network/fs problem, which might truncate the file
adding an EOF before the end and thus making the checksum fail..
But is it possible?
Or what else could it be?