51MB size limit with fopen in append mode

[Aaron Couts]

I have a program that writes to a log file. It's compiled on RH Linux 7.3
(Kernel 2.4.18-18.7). It's using fopen in append mode. When the file reaches
51200000 bytes in size, the program will no longer write to the file. When
this happens, fopen and fputs do not return an error. I've been researching
large file support for Linux, and it all has to do with the regular 2-gig file
size limit. If it's something obvious, sorry -- I'm a C newbie. Here's the
code snippet:

/* open logfile */
if ((lop = fopen(log, "a")) == NULL) {
fprintf(op, "\n%s: can't open %s for writing\n", prog, log);
fclose(op);
exit(1);
}

/* write To: line to logfile */
fputs(outputline, lop);
fclose(lop);
if (ferror(lop)) {
fprintf(op, "\n%s: error writing to %s\n", prog, log);
}

I'm all ears if anyone has any suggestions. Thanks.
-Aaron

 

[Kenneth Brody]

I have a program that writes to a log file. It's compiled on RH Linux 7.3
(Kernel 2.4.18-18.7). It's using fopen in append mode. When the file reaches
51200000 bytes in size, the program will no longer write to the file. When
this happens, fopen and fputs do not return an error. I've been researching
large file support for Linux, and it all has to do with the regular 2-gig file
size limit. If it's something obvious, sorry -- I'm a C newbie. Here's the
code snippet:

[...]

This is unrelated to the C language, but rather the O/S itself.

Try "man ulimit", and if that doesn't help, you could probably get a
better answer in one of the *nix newsgroups, such as within the
"comp.unix.*" or "comp.os.linux.*" trees.

--

+---------+----------------------------------+-----------------------------+
| Kenneth | kenbrody at spamcop.net | "The opinions expressed |
| J. | http://www.hvcomputer.com | herein are not necessarily |
| Brody | http://www.fptech.com | those of fP Technologies." |
+---------+----------------------------------+-----------------------------+

 

[Malcolm]

It's using fopen in append mode. When the file reaches
51200000 bytes in size, the program will no longer write to the file. >

When this happens, fopen and fputs do not return an error.This is the sort of thing that happens when you start straining the limits
of the platform. fputs() should return an error if the text is not
successfully written.
What you need to do, if you really do need 51MB files, is use lower-level
platform-specific calls to manage your IO.

 

[CBFalconer]

I have a program that writes to a log file. It's compiled on RH
Linux 7.3 (Kernel 2.4.18-18.7). It's using fopen in append mode.
When the file reaches 51200000 bytes in size, the program will no
longer write to the file. When this happens, fopen and fputs do
not return an error. I've been researching large file support
for Linux, and it all has to do with the regular 2-gig file size
limit. If it's something obvious, sorry -- I'm a C newbie.
Here's the code snippet:

/* open logfile */
if ((lop = fopen(log, "a")) == NULL) {
fprintf(op, "\n%s: can't open %s for writing\n", prog, log);
fclose(op);
exit(1);
}

/* write To: line to logfile */
fputs(outputline, lop);
fclose(lop);
if (ferror(lop)) {
fprintf(op, "\n%s: error writing to %s\n", prog, log);
}

I'm all ears if anyone has any suggestions. Thanks.

I see nothing outstandingly wrong, apart from the exit that should
be "exit(EXIT_FAILURE)" and the failure to check the returns from
fputs and fclose before the ferror check. I believe their is no
guarantee that ferror gets set on those errors.

 

[Eric Sosman]

I have a program that writes to a log file. It's compiled on RH Linux 7.3
(Kernel 2.4.18-18.7). It's using fopen in append mode. When the file reaches
51200000 bytes in size, the program will no longer write to the file. When
this happens, fopen and fputs do not return an error. I've been researching
large file support for Linux, and it all has to do with the regular 2-gig file
size limit. If it's something obvious, sorry -- I'm a C newbie. Here's the
code snippet:

/* open logfile */
if ((lop = fopen(log, "a")) == NULL) {
fprintf(op, "\n%s: can't open %s for writing\n", prog, log);
fclose(op);
exit(1);
}

/* write To: line to logfile */
fputs(outputline, lop);
fclose(lop);
if (ferror(lop)) {
fprintf(op, "\n%s: error writing to %s\n", prog, log);
}

The problem/limitation/whatever doesn't appear to have
anything to do with C per se, and you'll probably need to
look elsewhere for the solution. However, there's a bug in
the C snippet: You're calling ferror() on a FILE* that's
already been fclose()d -- which means, more or less, that
anything can happen. It's at least possible that "anything"
is a completely spurious error indication, and that you may
have no actual problem at all.

There are at least two ways to check for success of the
output functions: Either check the value returned by each
output-generating call (e.g., `if (fputs(...) == EOF)') or
just ignore the returned values and call ferror() at suitable
moments (prior to fclose()). Also, you should check the
value returned by fclose() itself. Yes, a close operation can
fail, and such a failure once caused me to make a plane trip
to an irate customer's site so I could grovel on his office
floor and pray forgiveness for my then employer.

 

[Nick Landsberg]

I have a program that writes to a log file. It's compiled on RH Linux 7.3
(Kernel 2.4.18-18.7). It's using fopen in append mode. When the file reaches
51200000 bytes in size, the program will no longer write to the file. When
this happens, fopen and fputs do not return an error. I've been researching
large file support for Linux, and it all has to do with the regular 2-gig file
size limit. If it's something obvious, sorry -- I'm a C newbie. Here's the
code snippet:

/* open logfile */
if ((lop = fopen(log, "a")) == NULL) {
fprintf(op, "\n%s: can't open %s for writing\n", prog, log);
fclose(op);
exit(1);
}

/* write To: line to logfile */
fputs(outputline, lop);
fclose(lop);
if (ferror(lop)) {
fprintf(op, "\n%s: error writing to %s\n", prog, log);
}

I'm all ears if anyone has any suggestions. Thanks.
-Aaron

The fopen() call, if the file exists, did not fail. If the
file exceeded some internal limit and you specified "a", then
one could argue that it should have failed, but I doubt
that any implemention of fopen() does that.

More importantly, you are checking ferror() AFTER
the fclose() call. Fclose() succeeded.

From my man page for fclose() (your mileage may vary)

"Upon successful completion 0 is returned. Otherwise,
EOF is returned and the global variable errno is set
to indicate the error. In either case
no further access to the stream is possible."

You should check ferror() before closing the stream.

(Then you should check errno to see why.)

 

[Sidney Cadot]

When this happens, fopen and fputs do not return an error.

This is the sort of thing that happens when you start straining the limits
of the platform.

Writing a file in excess of 51 MB is not "stretching the limits" by far.

fputs() should return an error if the text is not successfully written.

In particular, it should return an EOF if the write was unsuccesful.

What you need to do, if you really do need 51MB files, is use lower-level
platform-specific calls to manage your IO.

Pardon my french, but this is nonsense. Standard C handled big files
like a charm, even more so on the operating system in question. And
these files aren't even big, just - not small.

What the OP should do is run "quota" and "df", and see if this turns up
anything useful.

Best regards,

Sidney

 

[Jan Engelhardt]

What the OP should do is run "quota" and "df", and see if this turns up

anything useful.

Somewhere he might be able to get errno,
EDQUOT and ENOSPC.

 

[Aaron Couts]

I should have specified that there are no disk quotas on the system, and drive
space is not an issue. I've seen the same issue on two different machines.

Thanks for the suggestions. I'll check ferror after each f... call and see if
I can find out more info.

Also, this program is running as the user 'nobody'; perhaps the OS has some
special deal with that. I look into that as well.

 

[Nick Landsberg]

Eric Sosman wrote:

[SNIP]

Also, you should check the
value returned by fclose() itself. Yes, a close operation can
fail, and such a failure once caused me to make a plane trip
to an irate customer's site so I could grovel on his office
floor and pray forgiveness for my then employer.

Somehow, I'm heartened to know that I'm not the only one
who has had to go through this ordeal, Eric!
(And 'tweren't my code either!)

 

[Chris Torek]

Aaron Couts wrote:
[much snippage]
There are at least two ways to check for success of the
output functions: Either check the value returned by each
output-generating call (e.g., `if (fputs(...) == EOF)') or
just ignore the returned values and call ferror() at suitable
moments (prior to fclose()). Also, you should check the
value returned by fclose() itself. Yes, a close operation can
fail, and ...[/QUOTE]

Indeed.

For the lazy (which often includes me ) I recommend something
like the following sequence for "closing a file to which output has
been written":

const char *s;
...
s = NULL;
if (fflush(fp)) {
s = strerror(errno);
fprintf(stderr, "%s: error writing %s: %s\n", progname, filename, s);
}
if (s == NULL && ferror(fp))
fprintf(stderr, "%s: error writing %s (but I can't remember why)\n",
progname, filename);
if (fclose(fp)) {
s = strerror(errno);
fprintf(stderr, "%s: error closing %s: %s\n", progname, filename, s);
}

Although there is no guarantee that errno is set on a failing
fflush() or fclose() (or indeed any output function, if you are
not lazy and check for errors immediately), there is a significant
chance that it *was* set, and including the result from strerror()
can help figure out what went wrong. If you want to be especially
explicit and/or not trust the system to set errno, you might do
something like:

fprintf(stderr, "%s: error writing %s --\n"
"\tmost recent system error recorded was \"%s\",\n"
"\talthough this may have nothing to do with the problem.\n",
progname, filename, strerror(errno));

If you are "non-lazy" and check every output function for success,
you have a better chance of capturing a useful errno value. (Then
you also do not need the "ferror(fp)" test either.)

 

[Christopher Benson-Manica]

Yes, a close operation can
fail, and such a failure once caused me to make a plane trip
to an irate customer's site so I could grovel on his office
floor and pray forgiveness for my then employer.

Care to elaborate? Or is the memory too painful?

 

[Nick Landsberg]

Care to elaborate? Or is the memory too painful?

Maybe Eric doesn't care to elaborate, but I will.

What happens on the close() of an invalid file
descriptor (ANS: EBABF) and you did not check
the return value from close() ?

It probably indicates that somewhere in your code
the FD was inadvertantly modified, probably due to
poor naming conventions or some such. The FD
that was really open never gets closed. You have
a lot of files open. If you have to run for months at
a time without a reboot, sooner or later you run
out of O/S structures for all those open files
of yours. The result is left as a thought
experiment for the reader .

In essence this is equivalent to a "memory leak" in that
you "free()" the wrong resource and wind up using
an unnecessary resources until there are no more
left.

 

[Eric Sosman]

Care to elaborate? Or is the memory too painful?

I've told the tale before, so here's an abbreviated synopsis.
The program was a document-processing system, and when the user
saved the new version of an edited document the system did

stm = fopen("doc.tmp", "w");
if (stm == NULL) ...
while (more to write)
if (fwrite(buf, siz, cnt, stm) != cnt) ...
if (fflush(stm) != 0) ...
fclose (stm);
/* Document has been written successfully. */
remove ("doc.bak");
if (rename ("doc.doc", "doc.bak") != 0) ...
if (rename ("doc.tmp", "doc.doc") != 0) ...

Customer loaded his document, did a bunch of editing, and
then saved it. Then he did a little more editing -- possibly
a spell-check or something -- saved again, and went home. When
he came in the next morning the document file was truncated and
unusable, and so was the backup file. Since everything was
checked for errors, how did this happen?

Well, "everything" omitted the fclose(), which the author
apparently thought unnecessary to check -- after all, all the
"actual" output operations were checked, and fclose() is just
sort of a disconnector, right? Wrong. On the system at hand
(as on many systems) there were multiple layers in the I/O
system, and the fflush() did no more than push the data from
one layer to the next. Apparently some buffers still remained
in that further layer, and they weren't actually drained to
disk until fclose() tore down the entire connection to the file.
And when the further layer tried to drain its buffers to the
disk, it found that the user had exhausted his disk space
quota, and could allocate no more space to hold the data. The
further layer returned an error indication back to fclose(), but
the application ignored it.

Result from the first save: doc.bak holds a perfectly good
document without any edits, and doc.doc is a damaged file. After
the second save, both doc.bak and doc.doc are damaged files, and
no intact document -- with or without the user's edits -- remains
on the disk. As Murphy would have it, the user was in fact the
boss of the entire department, the guy who was negotiating with
us to buy upgrades and additional licenses ...

Heed the Sixth Commandment!

http://www.lysator.liu.se/c/ten-commandments.html

 

[Kenneth Brody]

I should have specified that there are no disk quotas on the system, and drive
space is not an issue. I've seen the same issue on two different machines.

[...]

Did you check the ulimit?

--

+---------+----------------------------------+-----------------------------+
| Kenneth | kenbrody at spamcop.net | "The opinions expressed |
| J. | http://www.hvcomputer.com | herein are not necessarily |
| Brody | http://www.fptech.com | those of fP Technologies." |
+---------+----------------------------------+-----------------------------+