Misleading error message when opening a file (on Windows XP SP 2)

Discussion in 'Python' started by Claudio Grondi, Aug 28, 2006.

  1. Here an example of what I mean
    (Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    large file):

    >>> f = file('veryBigFile.dat','r')
    >>> f = file('veryBigFile.dat','r+')


    Traceback (most recent call last):
    File "<pyshell#1>", line 1, in -toplevel-
    f = file('veryBigFile.dat','r+')
    IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'

    Is it a BUG or a FEATURE?

    Claudio Grondi
     
    Claudio Grondi, Aug 28, 2006
    #1
    1. Advertising

  2. In <ecu65e$12o$>, Claudio Grondi wrote:

    >
    > Here an example of what I mean
    > (Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    > large file):
    >
    > >>> f = file('veryBigFile.dat','r')
    > >>> f = file('veryBigFile.dat','r+')


    You mention the file size and gave a "speaking" name to that file -- does
    the file size matter?

    > Traceback (most recent call last):
    > File "<pyshell#1>", line 1, in -toplevel-
    > f = file('veryBigFile.dat','r+')
    > IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >
    > Is it a BUG or a FEATURE?


    It's the error number Windows returns for that operation.

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Aug 28, 2006
    #2
    1. Advertising

  3. Re: Misleading error message when opening a file (on Windows XP SP2)

    Marc 'BlackJack' Rintsch wrote:
    > In <ecu65e$12o$>, Claudio Grondi wrote:
    >
    >
    >>Here an example of what I mean
    >>(Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    >>large file):
    >>
    >> >>> f = file('veryBigFile.dat','r')
    >> >>> f = file('veryBigFile.dat','r+')

    >
    >
    > You mention the file size and gave a "speaking" name to that file -- does
    > the file size matter?

    Yes, it does.
    I haven't tested it yet, but I suppose 2 or 4 GByte threshold value.
    >
    >
    >>Traceback (most recent call last):
    >> File "<pyshell#1>", line 1, in -toplevel-
    >> f = file('veryBigFile.dat','r+')
    >>IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >>
    >>Is it a BUG or a FEATURE?

    >
    >
    > It's the error number Windows returns for that operation.

    So you just try to say:
    "it's not Python fault - it's just another bug of the damn Microsoft
    Windows operating system", right?

    Claudio Grondi
     
    Claudio Grondi, Aug 28, 2006
    #3
  4. Claudio Grondi

    Tim Peters Guest

    [Claudio Grondi]
    > Here an example of what I mean
    > (Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    > large file):
    >
    > >>> f = file('veryBigFile.dat','r')
    > >>> f = file('veryBigFile.dat','r+')

    >
    > Traceback (most recent call last):
    > File "<pyshell#1>", line 1, in -toplevel-
    > f = file('veryBigFile.dat','r+')
    > IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >
    > Is it a BUG or a FEATURE?


    Assuming the file exists and isn't read-only, I bet it's a Windows
    bug, and that if you open in binary mode ("r+b") instead I bet it goes
    away (this wouldn't be the first large-file text-mode Windows bug).
     
    Tim Peters, Aug 28, 2006
    #4
  5. Tim Peters wrote:

    > Assuming the file exists and isn't read-only, I bet it's a Windows
    > bug, and that if you open in binary mode ("r+b") instead I bet it goes
    > away (this wouldn't be the first large-file text-mode Windows bug).


    > dir bigfile.dat

    2006-08-28 11:46 5 000 000 000 bigfile.dat

    >>> f = file("bigfile.dat", "r")
    >>> f = file("bigfile.dat", "r+")

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory: 'bigfile.dat'

    >>> f = file("bigfile.dat", "rb")
    >>> f = file("bigfile.dat", "r+b")


    (typing f.read() here is a nice way to lock up the machine ;-)

    </F>
     
    Fredrik Lundh, Aug 28, 2006
    #5
  6. Re: Misleading error message when opening a file (on Windows XP SP2)

    Tim Peters wrote:
    > [Claudio Grondi]
    >
    >> Here an example of what I mean
    >> (Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    >> large file):
    >>
    >> >>> f = file('veryBigFile.dat','r')
    >> >>> f = file('veryBigFile.dat','r+')

    >>
    >> Traceback (most recent call last):
    >> File "<pyshell#1>", line 1, in -toplevel-
    >> f = file('veryBigFile.dat','r+')
    >> IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >>
    >> Is it a BUG or a FEATURE?

    >
    >
    > Assuming the file exists and isn't read-only, I bet it's a Windows
    > bug, and that if you open in binary mode ("r+b") instead I bet it goes
    > away (this wouldn't be the first large-file text-mode Windows bug).


    I knew already that 'r+b' fixes it. Yes, you have won the bet :) .

    I suppose, like you do, that because there is a difference between text
    and binary files on Windows and the text files are e.g. opened being
    buffered using a 32-bit buffer pointer, this fails on too large NTFS files.

    I could also imagine that Python tries to buffer the text file and fails
    because it uses the wrong pointer size when asking Windows for the
    content. I have not yet looked into the C-code of Python - any hint
    which file I should take a closer look at?
    Just curious to see for myself, that the bug is on the Windows side.

    Claudio Grondi
     
    Claudio Grondi, Aug 28, 2006
    #6
  7. Tim Peters wrote:

    >> Traceback (most recent call last):
    >> File "<pyshell#1>", line 1, in -toplevel-
    >> f = file('veryBigFile.dat','r+')
    >> IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >>
    >> Is it a BUG or a FEATURE?

    >
    > Assuming the file exists and isn't read-only, I bet it's a Windows
    > bug, and that if you open in binary mode ("r+b") instead I bet it goes
    > away (this wouldn't be the first large-file text-mode Windows bug).


    however, if you use the C level API, you get EINVAL (which presumably means
    that the CRT cannot open this file in text mode), not ENOENT. this is also true
    for older versions of Python:

    Python 2.1.1 (#20, Aug 23 2001, 11:27:17) [MSC 32 bit (Intel)] on win32
    >>> f = open("bigfile.dat")
    >>> f = open("bigfile.dat", "r+")

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    IOError: [Errno 22] Invalid argument: 'bigfile.dat'

    which probably means that this fix

    http://www.python.org/sf/538827

    is partially responsible for the misleading error message.

    (the cause of this seems to be that when you open a text file for updating, the
    CRT check if there's a chr(26) at the end of the file, but the 32-bit lseek API
    doesn't support seeking to positions larger than 2^31-2)

    </F>
     
    Fredrik Lundh, Aug 28, 2006
    #7
  8. Re: Misleading error message when opening a file (on Windows XP SP2)

    Fredrik Lundh wrote:
    > Tim Peters wrote:
    >
    >
    >>>Traceback (most recent call last):
    >>> File "<pyshell#1>", line 1, in -toplevel-
    >>> f = file('veryBigFile.dat','r+')
    >>>IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >>>
    >>>Is it a BUG or a FEATURE?

    >>
    >>Assuming the file exists and isn't read-only, I bet it's a Windows
    >>bug, and that if you open in binary mode ("r+b") instead I bet it goes
    >>away (this wouldn't be the first large-file text-mode Windows bug).

    >
    >
    > however, if you use the C level API, you get EINVAL (which presumably means
    > that the CRT cannot open this file in text mode), not ENOENT. this is also true
    > for older versions of Python:
    >
    > Python 2.1.1 (#20, Aug 23 2001, 11:27:17) [MSC 32 bit (Intel)] on win32
    >
    >>>>f = open("bigfile.dat")
    >>>>f = open("bigfile.dat", "r+")

    >
    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > IOError: [Errno 22] Invalid argument: 'bigfile.dat'
    >
    > which probably means that this fix
    >
    > http://www.python.org/sf/538827
    >
    > is partially responsible for the misleading error message.
    >
    > (the cause of this seems to be that when you open a text file for updating, the
    > CRT check if there's a chr(26) at the end of the file, but the 32-bit lseek API
    > doesn't support seeking to positions larger than 2^31-2)
    >
    > </F>


    Using MSVC++ .NET 2003 compiler (if I did it all the right way):

    fstm = fopen("bigfile.dat","r+");
    if(fstm == 0) { printf( " ErrNo: %i \n", errno ); }
    // ^-- prints :
    // on "r+" with too large file: 22 (EINVAL-Invalid argument)
    // on non-existing file : 2 (ENOENT-no such file)
    // on bad mode string spec. : 0 (??? why not EINVAL ...)

    So there _is_ a way to distinguish the different problems occurred while
    opening the file. The error message comes from Python (errnomodule.c),
    not from Windows(errno.h). Concluding from this it becomes evident for
    me, that this misleading error message is Python fault (even if
    originated by misleading errno values set after fopen in the MSVC++
    environment and Windows), right?
    Probably also in Python 2.5?

    Claudio Grondi
     
    Claudio Grondi, Aug 28, 2006
    #8
  9. Claudio Grondi

    Georg Brandl Guest

    Re: Misleading error message when opening a file (on Windows XP SP2)

    Claudio Grondi wrote:
    > Tim Peters wrote:
    >> [Claudio Grondi]
    >>
    >>> Here an example of what I mean
    >>> (Python 2.4.2, IDLE 1.1.2, Windows XP SP2, NTFS file system, 80 GByte
    >>> large file):
    >>>
    >>> >>> f = file('veryBigFile.dat','r')
    >>> >>> f = file('veryBigFile.dat','r+')
    >>>
    >>> Traceback (most recent call last):
    >>> File "<pyshell#1>", line 1, in -toplevel-
    >>> f = file('veryBigFile.dat','r+')
    >>> IOError: [Errno 2] No such file or directory: 'veryBigFile.dat'
    >>>
    >>> Is it a BUG or a FEATURE?

    >>
    >>
    >> Assuming the file exists and isn't read-only, I bet it's a Windows
    >> bug, and that if you open in binary mode ("r+b") instead I bet it goes
    >> away (this wouldn't be the first large-file text-mode Windows bug).

    >
    > I knew already that 'r+b' fixes it. Yes, you have won the bet :) .
    >
    > I suppose, like you do, that because there is a difference between text
    > and binary files on Windows and the text files are e.g. opened being
    > buffered using a 32-bit buffer pointer, this fails on too large NTFS files.
    >
    > I could also imagine that Python tries to buffer the text file and fails
    > because it uses the wrong pointer size when asking Windows for the
    > content. I have not yet looked into the C-code of Python - any hint
    > which file I should take a closer look at?


    That would be Objects/fileobject.c. And no, on just open()ing the file,
    Python does nothing more than fopen() (or some Windows equivalent).

    Georg
     
    Georg Brandl, Aug 28, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. j
    Replies:
    5
    Views:
    425
    Samuel Barber
    Jul 27, 2003
  2. Brian Kelley

    Misleading Python error message

    Brian Kelley, Nov 19, 2003, in forum: Python
    Replies:
    4
    Views:
    378
    Dennis Lee Bieber
    Nov 21, 2003
  3. jkn
    Replies:
    5
    Views:
    514
    Fredrik Lundh
    Nov 22, 2006
  4. Erik Johnson
    Replies:
    4
    Views:
    367
  5. Roy Smith

    Misleading error message of the day

    Roy Smith, Dec 8, 2011, in forum: Python
    Replies:
    37
    Views:
    564
    Lie Ryan
    Dec 10, 2011
Loading...

Share This Page