file size limit exceeded

[Peter Ballard]

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.

Regards,

Peter Ballard
Adelaide, AUSTRALIA
(e-mail address removed)
http://www.ozemail.com.au/~pballard/

 

[Jack Klein]

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just wondering.

Please use a proper signature block, like mine, that begins with a
line with the three characters "-- ", so proper newsreaders will
automatically snip it when replying.

As to file size limitations, they have nothing at all to do with the C
standard and everything to do with your compiler and/or operating
system. You need to ask in a group like

--
Jack Klein
Home: http://JK-Technology.Com
FAQs for
comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html
comp.lang.c++ http://www.parashift.com/c++-faq-lite/
alt.comp.lang.learn.c-c++ ftp://snurse-l.org/pub/acllc-c++/faq

 

[Glen Herrmannsfeldt]

Hi all,

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here?

(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().

The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.

-- glen

 

[Dan Pop]

(snip)

The C library routines allow fseek() calls to be made on the file. The
system doesn't know that you aren't doing any fseek().

So what? fseek() is allowed to fail.

The position argument to fseek() is an int. To prevent problems that could
occur, the system limits file size.

Nonsense! That's what fsetpos() is for.

Dan

 

[Dan Pop]

# Now since it works, it's not really urgent. But I'm curious: what's
# going on here? I've read that the "file size limit exceeded" error is
# due to my OS (which is Linux, 2.4 I think), but then why should the OS
# care whether the file is opened and closed inside my C program (when
# it didn't work) or in bash (when it did work)? Is the problem due to
# some pointer size limit in my machine's implementation of C (the
# program was compiled using gcc)? Or something else?

Are you sure it's the kernel or stdio? You can bypass stdio and use
open and write calls to check if it also fails at 2GB.

How do you suppose stdio is actually performing its I/O?

Next time, engage your brain before posting.

Dan

 

[Dan Pop]

I've got a C program which outputs all its data using a statement of
the form:

putchar(ch, outfile);

This has worked fine for years until it had to output more than 2GB of
data today, and I got a "file size limit exceeded" error.

On a whim I tried setting 'outfile' to 'stdout' (and piped output
using the bash '> outfile'), and it worked.

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

Just a guess. Your system has large file support, but it is not enabled
by default. When fopen() calls open(), it doesn't specify whatever magic
is necessary for enabling the large file support. When bash opens a file,
it does.

The magic is actually documented in the open(2) man page.

Dan

 

[Randy Howard]

Now since it works, it's not really urgent. But I'm curious: what's
going on here? I've read that the "file size limit exceeded" error is
due to my OS (which is Linux, 2.4 I think), but then why should the OS
care whether the file is opened and closed inside my C program (when
it didn't work) or in bash (when it did work)? Is the problem due to
some pointer size limit in my machine's implementation of C (the
program was compiled using gcc)? Or something else?

That's OT for this newsgroup (comp.os.linux.development.* for example)
however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

 

[Peter Ballard]

That's OT for this newsgroup (comp.os.linux.development.* for example)

Probably. But in my defence, my problem was unique to my C program.
(bash was fine making >2GB; as were test programs in two other
languages which I tried since my 1st post).

however, a google search for "LFS" "Large file summit", 2GB, etc.
will probably lead you to the right answer as well.

Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html

--
Regards,

Peter Ballard
Adelaide, AUSTRALIA
(e-mail address removed)
http://www.ozemail.com.au/~pballard/

 

[Dan Pop]

(snip)


I thought it was pretty strange, too. OK, the way I understood it is that
if the library let programs write files over 2GB they could overwrite data
as fseek() would wrap. Rather then let files accidentally overwrite data,
they generate a fatal error. There is a bit somewhere that indicates that
the program actually can handle files of 2GB or more, in which case it
succeeds.

Now, on the OS that I found this on, output redirect would still fail
because redirecting to a file allows fseek(). If the output was a
non-seekable pipe then it wouldn't fail, and one could pipe the output to
cat, redirect the output of cat, and write large files.

Very strange, I agree!

It's not strange at all, once you understand that the O_LARGEFILE flag
of the open() function is not set by default on certain platforms. The
behaviour of each application depends on whether it sets it or not when
opening the file. Of course, portable code doesn't have this option:
if fopen() doesn't do it, you're stuck to "small" files.

Dan

 

[Randy Howard]

Probably. But in my defence, my problem was unique to my C program.

It could have also happened with any language using 32bit file offsets.
Still doesn't make it a c.l.c. issue.

Thanks to all who responded. This URL seems to cover the issue fairly
well:

http://www.suse.de/~aj/linux_lfs.html

Indeed. With the correct platform-dependant invocations, you can make
Linux C programs generate (or interact with) files over several terabytes.
Again, that's not really the c.l.c. problem domain.