IOError:[Errno 27] File too large

Discussion in 'Python' started by ch.valderanis@gmail.com, Mar 13, 2013.

  1. Guest

    Hi,

    Relatively newcomer here.
    The following code fails with the above error:
    python version used 2.6.2 under linux

    filestring='somestring'
    for files in glob.glob('*'):
    f2=open(files.replace('.xml','.sub'),'w')
    f2.write(filestring+files)
    f2.close()


    The glob commands returns around 10k files.
    I want to create 10k new files. the filename should have a different extension (sub instead of xml)and the content of the new file should be some filestring with the addition of the original filename.

    I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?

    Any other comments,remarks welcomed.

    Thanks,
    Makis
    , Mar 13, 2013
    #1
    1. Advertising

  2. On Wed, 13 Mar 2013 09:53:17 -0700, ch.valderanis wrote:

    > Hi,
    >
    > Relatively newcomer here.
    > The following code fails with the above error: python version used 2.6.2
    > under linux


    Which part of the code fails? Please copy and paste the entire traceback,
    starting with the line "Traceback (most recent call last)" and ending
    with the complete error message.

    Which file does it fail on?


    > filestring='somestring'
    > for files in glob.glob('*'):
    > f2=open(files.replace('.xml','.sub'),'w')
    > f2.write(filestring+files)
    > f2.close()


    Some other comments:

    You use the name "files" to represent a single filename, rather than
    multiple files. That is misleading, a poor choice of name.

    "f2" is also a poor name.

    You try to change the file extension using string.replace method. This is
    risky, and will fail if some file has ".xml" in the filename apart from
    the file extension. Worse, if you have a file *without* ".xml", your code
    will over-write that file. Better to use the os.path.splitext function
    for this.

    Is filestring really a constant string like in the snippet above? Is
    there any change that it could be an enormous string?


    Re-writing your code snippet:


    import os
    filestring = 'somestring'
    for filename in glob.glob('*.xml'):
    newname = os.path.splitext(filename)[0] + '.sub'
    newfile = open(newname, 'w')
    newfile.write(filestring + filename)
    newfile.close()



    > I am pretty sure that the files is less than 1kB.


    "Pretty sure"? You need to be 100% certain. We can't tell you whether
    this is the case or not.



    > Is there another reason for the operation to fail?


    We don't know which operation has failed. That's why you need to show the
    complete traceback.




    --
    Steven
    Steven D'Aprano, Mar 13, 2013
    #2
    1. Advertising

  3. On Wed, 13 Mar 2013 09:53:17 -0700 (PDT),
    declaimed the following in gmane.comp.python.general:

    > Hi,
    >
    > Relatively newcomer here.
    > The following code fails with the above error:
    > python version used 2.6.2 under linux
    >
    > filestring='somestring'
    > for files in glob.glob('*'):
    > f2=open(files.replace('.xml','.sub'),'w')


    "Danger Will Robinson, Danger"

    What do you expect to have happen for files that did NOT have ".xml"
    somewhere in the name. If you only want to do this for *.xml you should
    probably ask for that in the glob() call.

    Also, the standard library has functions for splitting file names
    into prefix and extension...

    > f2.write(filestring+files)
    > f2.close()


    <snip>

    > I am pretty sure that the files is less than 1kB. Is there another reason for the operation to fail?
    >
    > Any other comments,remarks welcomed.
    >

    Provide the full traceback, so we can confirm what line produced the
    error.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Mar 13, 2013
    #3
  4. Guest

    Dear Steven,

    Thank you very much both for your answer and of course your comments. They are taken into account.
    I found out that when I touch FILENAME.sub in the command line, I get the same error. So I guess it isn't a problem with the language but rather deeper. I will solve this first before I come back.
    Just for completeness:

    > Which part of the code fails? Please copy and paste the entire traceback,
    > starting with the line "Traceback (most recent call last)" and ending
    > with the complete error message.

    The code is corrected according to your advice:
    Here it the error:

    Traceback (most recent call last):
    File "createsubmitfiles.py", line 12, in <module>
    newfile=open(newname,'w')
    IOError: [Errno 27] File too large: 'FILENAME.sub;'

    I have changed the actual filename reported by the traceback.
    FILENAME.xml does exist before the code is run
    FILENAME.sub is not created
    FILENAME has a length of 160 characters
    I do have space to write all the files in the directory

    > Is filestring really a constant string like in the snippet above? Is
    > there any change that it could be an enormous string?

    Yes. It is a constant string


    > > I am pretty sure that the files is less than 1kB.

    > "Pretty sure"? You need to be 100% certain. We can't tell you whether
    > this is the case or not.

    When I do an ls -l before I run the code and after the code has failed, all I see are files with size <1kB.

    Makis
    , Mar 13, 2013
    #4
  5. On Wed, 13 Mar 2013 15:35:19 -0700, ch.valderanis wrote:

    > Traceback (most recent call last):
    > File "createsubmitfiles.py", line 12, in <module>
    > newfile=open(newname,'w')
    > IOError: [Errno 27] File too large: 'FILENAME.sub;'
    >
    > I have changed the actual filename reported by the traceback.
    > FILENAME.xml does exist before the code is run FILENAME.sub is not
    > created
    > FILENAME has a length of 160 characters I do have space to write all the
    > files in the directory


    You are not trying to create a file called "FILENAME.sub". You are trying
    to create a file called "FILENAME.sub;", notice the semi-colon.

    Taking a wild guess, I think that you are using a Samba share on a Linux
    server. A file "FILENAME.xml;" was accidentally creating on this share
    from the Linux filesystem layer, since Linux will allow you to use
    semicolons in file names. But samba enforces the same restrictions as
    Windows, so when you access the file system through sambda, it gives an
    error when you try to create a new file "FILENAME.sub;".

    This is a wild guess, and could be completely wrong. Please come back and
    tell us the solution when you have solved it.



    Good luck!



    --
    Steven
    Steven D'Aprano, Mar 14, 2013
    #5
  6. Guest


    > Taking a wild guess, I think that you are using a Samba share on a Linux
    >
    > server. A file "FILENAME.xml;" was accidentally creating on this share
    >
    > from the Linux filesystem layer, since Linux will allow you to use
    >
    > semicolons in file names. But samba enforces the same restrictions as
    >
    > Windows, so when you access the file system through sambda, it gives an
    >
    > error when you try to create a new file "FILENAME.sub;".
    >
    > This is a wild guess, and could be completely wrong. Please come back and
    >
    > tell us the solution when you have solved it.


    So, let me report the solution.
    All the files were in an afs filesystem. afs has a limit for the total length of the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
    In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.

    Makis
    , Mar 14, 2013
    #6
  7. Dave Angel Guest

    On 03/14/2013 04:12 PM, wrote:
    >
    >> Taking a wild guess, I think that you are using a Samba share on a Linux
    >>
    >> server. A file "FILENAME.xml;" was accidentally creating on this share
    >>
    >> from the Linux filesystem layer, since Linux will allow you to use
    >>
    >> semicolons in file names. But samba enforces the same restrictions as
    >>
    >> Windows, so when you access the file system through sambda, it gives an
    >>
    >> error when you try to create a new file "FILENAME.sub;".
    >>
    >> This is a wild guess, and could be completely wrong. Please come back and
    >>
    >> tell us the solution when you have solved it.

    >
    > So, let me report the solution.
    > All the files were in an afs filesystem. afs has a limit for the total length of the filenames per directory. for more details take a look at http://www.inf.ed.ac.uk/systems/AFS/topten.html#Tip13
    > In my case 10k files with ~160characters per filename were just hitting the limit. The error message is basically misleading.
    >
    > Makis
    >


    Aha! So the file that was too large was the directory file.

    --
    DaveA
    Dave Angel, Mar 15, 2013
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jay Donnell

    IOError: [Errno 32] Broken pipe

    Jay Donnell, Jun 28, 2004, in forum: Python
    Replies:
    2
    Views:
    8,361
    Donn Cave
    Jun 29, 2004
  2. Nigel King

    IOError: [Errno 32] Broken pipe

    Nigel King, Sep 29, 2004, in forum: Python
    Replies:
    3
    Views:
    9,804
    Vinay Sajip
    Oct 11, 2004
  3. Replies:
    0
    Views:
    297
  4. Marco
    Replies:
    0
    Views:
    732
    Marco
    Feb 6, 2007
  5. Marco
    Replies:
    7
    Views:
    4,499
    Gabriel Genellina
    Feb 17, 2007
Loading...

Share This Page