File Size Limit Exceeded - How to handle work around this?

Discussion in 'C Programming' started by eastcoastguyz, Dec 13, 2006.

  1. I wrote a simple program to continue to create a very large file (on
    purpose), and even though there is plenty of disk space on that device
    the program aborted with the error message "File Size Limit Exceeded".
    The file size was 2147483647. I checked ulimit -a and its set to
    unlimited.

    Is this a compiler issue? I would like to see a C code example of how
    to increase the limit or make it unlimited (if that is a wise thing to
    do).

    Thanks in advance!
    eastcoastguyz, Dec 13, 2006
    #1
    1. Advertising

  2. In article <>,
    eastcoastguyz <> wrote:
    >I wrote a simple program to continue to create a very large file (on
    >purpose), and even though there is plenty of disk space on that device
    >the program aborted with the error message "File Size Limit Exceeded".
    >The file size was 2147483647. I checked ulimit -a and its set to
    >unlimited.


    >Is this a compiler issue?


    Probably not.

    >I would like to see a C code example of how
    >to increase the limit or make it unlimited (if that is a wise thing to
    >do).


    It is probably an operating system limitation (or a disk quota
    limitation). ulimit (which is not part of C) with -a set to
    unlimited, just means that the operating system will allow you to
    write files as big as is supported by that particular file system.

    Whether your operating system supports larger files at all would
    be OS specific, as would be any special means to create such files.
    This is a matter that should be taken to a resource that deals
    with your specific OS. It is -possible- that they will tell you
    there to change some flags to your compiles, but that would be
    for deep OS implementation reasons, not for reasons directly related
    to standard C.
    --
    If you lie to the compiler, it will get its revenge. -- Henry Spencer
    Walter Roberson, Dec 13, 2006
    #2
    1. Advertising

  3. eastcoastguyz

    Guest

    eastcoastguyz wrote:
    > I wrote a simple program to continue to create a very large file (on
    > purpose), and even though there is plenty of disk space on that device
    > the program aborted with the error message "File Size Limit Exceeded".
    > The file size was 2147483647. I checked ulimit -a and its set to
    > unlimited.
    >
    > Is this a compiler issue? I would like to see a C code example of how
    > to increase the limit or make it unlimited (if that is a wise thing to
    > do).
    >
    > Thanks in advance!


    Write the file in blocks and maintain your own chain or use a database,
    which will already have done that
    , Dec 13, 2006
    #3
  4. I forgot to mention, the OS is Linux with the latest version of CentOS
    using 'cc'.

    eastcoastguyz wrote:
    > I wrote a simple program to continue to create a very large file (on
    > purpose), and even though there is plenty of disk space on that device
    > the program aborted with the error message "File Size Limit Exceeded".
    > The file size was 2147483647. I checked ulimit -a and its set to
    > unlimited.
    >
    > Is this a compiler issue? I would like to see a C code example of how
    > to increase the limit or make it unlimited (if that is a wise thing to
    > do).
    >
    > Thanks in advance!
    eastcoastguyz, Dec 13, 2006
    #4
  5. eastcoastguyz

    Michael Guest

    > I wrote a simple program to continue to create a very large file (on
    > purpose), and even though there is plenty of disk space on that device
    > the program aborted with the error message "File Size Limit Exceeded".
    > The file size was 2147483647. I checked ulimit -a and its set to
    > unlimited.
    >
    > Is this a compiler issue? I would like to see a C code example of how
    > to increase the limit or make it unlimited (if that is a wise thing to
    > do).


    It's been a long time, though, so this advice may be dated.

    A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
    file sizes were just starting to catch on), we used to have this same
    problem. Back then, you had to compile with some special symbol
    defined, LARGE_FILES or something like that. That caused the typical
    file size types (offset_t and the like) to be 64 bit. You might find
    something like that in the compiler documentation.

    Of course, to narrow it to OS or C, you could have your program write
    to stdout and redirect to a file. If it still doesn't work, you have
    an OS problem. And if it isn't C, it could be your OS kernel, or your
    shell, or your filesystem (e.g., NFS) that has some kind of limit in
    it.

    Michael
    Michael, Dec 13, 2006
    #5
  6. Michael wrote:
    > > I wrote a simple program to continue to create a very large file (on
    > > purpose), and even though there is plenty of disk space on that device
    > > the program aborted with the error message "File Size Limit Exceeded".
    > > The file size was 2147483647. I checked ulimit -a and its set to
    > > unlimited.
    > >
    > > Is this a compiler issue? I would like to see a C code example of how
    > > to increase the limit or make it unlimited (if that is a wise thing to
    > > do).

    >
    > It's been a long time, though, so this advice may be dated.
    >
    > A long time ago, when the Linux 2.4 kernel was new (i.e., when 64-bit
    > file sizes were just starting to catch on), we used to have this same
    > problem. Back then, you had to compile with some special symbol
    > defined, LARGE_FILES or something like that. That caused the typical
    > file size types (offset_t and the like) to be 64 bit. You might find
    > something like that in the compiler documentation.
    >
    > Of course, to narrow it to OS or C, you could have your program write
    > to stdout and redirect to a file. If it still doesn't work, you have
    > an OS problem. And if it isn't C, it could be your OS kernel, or your
    > shell, or your filesystem (e.g., NFS) that has some kind of limit in
    > it.
    >
    > Michael


    Thanks for the suggestion. I made another version of this C program,
    where it writes to standard output and piped the output to a file. It
    had no problem creating a file size over 3GB, exceeding the 2GB limit I
    encountered.
    This system is using gcc version 3.4.6 20060404 (Red Hat 3.4.6-3) which
    is what came with the latest version of CentOS. The kernel is:
    [root@localhost proc]# cat /proc/version
    Linux version 2.6.9-42.0.3.EL (buildsvn@build-i386) (gcc version 3.4.6
    20060404
    eastcoastguyz, Dec 13, 2006
    #6
  7. eastcoastguyz

    Guest

    Hello,

    > > > I wrote a simple program to continue to create a very large file (on
    > > > purpose), and even though there is plenty of disk space on that device
    > > > the program aborted with the error message "File Size Limit Exceeded".
    > > > The file size was 2147483647. I checked ulimit -a and its set to
    > > > unlimited.


    Try to compile your program with the following flags:
    -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

    HTH,
    Loic.
    , Dec 13, 2006
    #7
  8. On Wed, 13 Dec 2006, wrote:

    > Hello,
    >
    >>>> I wrote a simple program to continue to create a very large file (on
    >>>> purpose), and even though there is plenty of disk space on that device
    >>>> the program aborted with the error message "File Size Limit Exceeded".
    >>>> The file size was 2147483647. I checked ulimit -a and its set to
    >>>> unlimited.

    >
    > Try to compile your program with the following flags:
    > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64


    Probably he meant:

    -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
    Kohn Emil Dan, Dec 13, 2006
    #8
  9. Kohn Emil Dan wrote:
    > On Wed, 13 Dec 2006, wrote:
    >
    > > Hello,
    > >
    > >>>> I wrote a simple program to continue to create a very large file (on
    > >>>> purpose), and even though there is plenty of disk space on that device
    > >>>> the program aborted with the error message "File Size Limit Exceeded".
    > >>>> The file size was 2147483647. I checked ulimit -a and its set to
    > >>>> unlimited.

    > >
    > > Try to compile your program with the following flags:
    > > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

    >
    > Probably he meant:
    >
    > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64


    Yes! This worked. Thanks to everyone who posted in response to my
    request for help.

    Does anyone know if it is required to do these compiler options
    regardless of the hardware the same OS runs on? Thanks!
    eastcoastguyz, Dec 13, 2006
    #9
  10. eastcoastguyz

    Nelu Guest

    eastcoastguyz wrote:
    > Kohn Emil Dan wrote:
    >> On Wed, 13 Dec 2006, wrote:
    >>
    >>> Hello,
    >>>
    >>>>>> I wrote a simple program to continue to create a very large file (on
    >>>>>> purpose), and even though there is plenty of disk space on that device
    >>>>>> the program aborted with the error message "File Size Limit Exceeded".
    >>>>>> The file size was 2147483647. I checked ulimit -a and its set to
    >>>>>> unlimited.
    >>> Try to compile your program with the following flags:
    >>> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS 64

    >> Probably he meant:
    >>
    >> -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

    >
    > Yes! This worked. Thanks to everyone who posted in response to my
    > request for help.
    >
    > Does anyone know if it is required to do these compiler options
    > regardless of the hardware the same OS runs on? Thanks!
    >


    <OT>
    I think the only macro you need defined is _GNU_SOURCE which will
    enable a number of macros and the ...LARGEFILE64... ones are
    among them. I think this is the preferred way to do this. I
    couldn't find _GNU_SOURCE in my header files on 64 bit Gentoo
    </OT>
    but it may be because I didn't pay too much attention. You
    shouldn't have posted the question here and you should not
    consider advices received from this group. The standard C doesn't
    know about 64 vs 32 bit, files larger than 2 GB or specific
    extensions. This is a question for a group that deals with the
    combination of compiler/OS that you're using.

    --
    Ioan - Ciprian Tandau
    tandau _at_ freeshell _dot_ org (hope it's not too late)
    (... and that it still works...)
    Nelu, Dec 14, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Eric Sabine
    Replies:
    0
    Views:
    336
    Eric Sabine
    Aug 25, 2003
  2. Brian Piotrowski

    Response Buffer Limit Exceeded Help Needed

    Brian Piotrowski, Aug 27, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    3,082
    Brian Piotrowski
    Aug 27, 2004
  3. Peter Ballard

    file size limit exceeded

    Peter Ballard, Jul 31, 2003, in forum: C Programming
    Replies:
    9
    Views:
    4,366
    Randy Howard
    Aug 2, 2003
  4. Replies:
    1
    Views:
    1,076
    Victor Bazarov
    Jun 28, 2005
  5. Replies:
    0
    Views:
    771
Loading...

Share This Page