looking for statefull snprintf

D

dimka

Hello

Here is my situation: I have a char buf[1024]; that is used as a buffer

for a log output. Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the

arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

Thanks
 
A

Andrew Poelstra

Hello

Here is my situation: I have a char buf[1024]; that is used as a buffer

for a log output. Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the

arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

I'm not sure about libraries, but you could check out the code for
Chuck Falconers ggets() code (Google), or some of the stuff in my
ioutils package (see my sig for link).

Chuck's code will give you some idea on how to read in "unlimited"
amounts of data, and mine has some minimal input parsing you might
want to check out.
 
B

Ben Pfaff

dimka said:
Here is my situation: I have a char buf[1024]; that is used as a buffer
for a log output. Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the
arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

The usual solution is allocate a bigger buffer and try again,
instead of trying to restart from the point where the first try
left off, which as you say is not well supported.

The snprintf function returns the size of the buffer you need
(although you need to add 1 to make room for a terminating null).
 
A

Ancient_Hacker

dimka said:
Hello

Here is my situation: I have a char buf[1024]; that is used as a buffer

memory is selling nowdays for about $60 a gigabyte. By my calculations
you're allocating
$0.006144 of memory for this buffer. If you're paid $5 an hour, and
it took you 2.5 minutes to type this message, you've wasted $0.208333
just typing in this message. For that money, you could have bought 30
times the memory you allocated.

Hows about you either:

(1) Bump up the memory buffer size. Assuming it's a local variable,
it's not going to tie up the memory for more than a few microseconds.

(2) Write the data directly to the file with printf(), then there's no
need for the temporary buffer.

(3) Estimate the amount of memory needed beforehand, then add a penny.
 
C

CBFalconer

Ben said:
dimka said:
Here is my situation: I have a char buf[1024]; that is used as a
buffer for a log output. Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format
or the arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

The usual solution is allocate a bigger buffer and try again,
instead of trying to restart from the point where the first try
left off, which as you say is not well supported.

The snprintf function returns the size of the buffer you need
(although you need to add 1 to make room for a terminating null).
From N869:

7.19.6.5 The snprintf function

Synopsis

[#1]

#include <stdio.h>
int snprintf(char * restrict s, size_t n,
const char * restrict format, ...);

Description

[#2] The snprintf function is equivalent to fprintf, except
that the output is written into an array (specified by
argument s) rather than to a stream. If n is zero, nothing
is written, and s may be a null pointer. Otherwise, output
characters beyond the n-1st are discarded rather than being
written to the array, and a null character is written at the
end of the characters actually written into the array. If
copying takes place between objects that overlap, the
behavior is undefined.

Returns

[#3] The snprintf function returns the number of characters
that would have been written had n been sufficiently large,
not counting the terminating null character, or a negative
value if an encoding error occurred. Thus, the null-
terminated output has been completely written if and only if
the returned value is nonnegative and less than n.

Taking advantage of this you can pretest for buffer overflow:

templgh = snprintf(NULL, 0, format, whatever);
if (templgh + bufindex >= sizeof(buffer) {
flushbuf(buffer); bufindex = 0;
}
bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943
 
B

Ben Pfaff

CBFalconer said:
Ben said:
The usual solution is allocate a bigger buffer and try again,
instead of trying to restart from the point where the first try
left off, which as you say is not well supported.

The snprintf function returns the size of the buffer you need
(although you need to add 1 to make room for a terminating null).
[...]

Taking advantage of this you can pretest for buffer overflow:

templgh = snprintf(NULL, 0, format, whatever);
if (templgh + bufindex >= sizeof(buffer) {
flushbuf(buffer); bufindex = 0;
}
bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

You sure, but it's likely to be faster to call snprintf just once
in the common case (and occasionally a second time), instead of
always calling it twice.

Also, I would make the code above a little smarter, so that it
can expand the buffer in case flushing it isn't sufficient to
make enough room.
 
D

dimka

This is funny :) Everyone can cite the standard. But this is EXACTLY
the code that
does NOT work.
 
B

Ben Pfaff

dimka said:
templgh = snprintf(NULL, 0, format, whatever);
if (templgh + bufindex >= sizeof(buffer) {
flushbuf(buffer); bufindex = 0;
}
bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

This is funny :) Everyone can cite the standard. But this is EXACTLY
the code that
does NOT work.

Perhaps you are using a pre-standard implementation of snprintf.
Some of these implementations just returned -1 when the string
wouldn't fit in the buffer. Then the caller is left to guess how
much room is needed and try again.
 
D

dimka

Well, let me explain the problem again.
Lets assume that the buffer is empty, thus no need to flush it.
then

rc = snprintf( buffer, sizeof(buffer)-1, "%s", str );

Works pretty well. No buffer overrun, in the new standard
rc == strlen(str), in old standard rc == -1, thus I have a way
to check that there was the potential overrun.

The problem arises when strlen(str) > sizeof(buffer).
In this situation the buffer will contain trunkated version
of str, I have the information that the string was
trunkated, but no way to say "hey, I have flushed the
buffer, please continue from the place where you
stopped". There must to be an alternative implementation
of snprintf() to do exactly what I want.


Ben said:
dimka said:
templgh = snprintf(NULL, 0, format, whatever);
if (templgh + bufindex >= sizeof(buffer) {
flushbuf(buffer); bufindex = 0;
}
bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

This is funny :) Everyone can cite the standard. But this is EXACTLY
the code that
does NOT work.

Perhaps you are using a pre-standard implementation of snprintf.
Some of these implementations just returned -1 when the string
wouldn't fit in the buffer. Then the caller is left to guess how
much room is needed and try again.
--
Peter Seebach on C99:
"[F]or the most part, features were added, not removed. This sounds
great until you try to carry a full-sized printout of the standard
around for a day."
 
W

websnarf

dimka said:
Here is my situation: I have a char buf[1024]; that is used as a buffer
for a log output.

You should immediately cringe whenever you find yourself in situations
where you are declaring large arrays like that, basically in hopes that
1024 is close enough to infinity.
[...] Normally, I do:

rc = snprintf( buf, sizeof(buf), some_format, some_args );

This makes the formatting my output and then I flush the buffer.
In 90% of cases this works well, exept the cases when the format or the
arguments require more then 1024 bytes of formatted input.
There is no easy way to re-run snprintf() form 'last' position.

Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).
Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

Another approach, of course, is to go find some open source snprintf's
out there and hack on them to make them do what you want. There are
numerous such snprintf packages out there. However, its also possible
to write your own. Either way, what you're going to want to do is to
change interface to something like:

long vpprintf (long (*output) (const char * fragment, int sz,
void * ctx),
void * ctx, const char * fmt, va_list arg);

The idea is that you would incrementally generate the formatted output
into fragments that you would then emit to the output() function
pointer in pieces (with the length passed in; this is required so that
you can output %c, with a value of '\0'). As a bonus, you would make
it so that if output() returned a negative number vpprintf would stop,
and re-return this value (to do some sort of error propogation.)
Otherwise you would return the total positive number of characters
"written" to the output callback function. Such a solution would
always use a finite amount of memory but be able to process an
arbitrarily long amount of formatted output.

Keep in mind that its not impossible to write this code from scratch
yourself. Primarily you have to break down the formatting and
arguments (via va_arg) yourself, but for the actual conversion you can
generally fall back on just plain old sprintf(). Obviously, you would
detect formatting widths beyond a certain size, and perform some
specially slicing and dicing and iterate it through a single reasonably
sized auto buffer. So there is not need to solve the difficult task of
converting floating point a to string or similar kinds of things.

If you write this code, you will basically be able to solve any problem
you ever have with C-style formatted output just using this interface.
What this says about the C standard committee (i.e., the fact that this
function or equivalent (i.e., allow for FILE *'s to wrap function
callbacks) doesn't already exist) I will leave for you to decide.
 
B

Ben Pfaff

dimka said:
Well, let me explain the problem again.
Lets assume that the buffer is empty, thus no need to flush it.
then

rc = snprintf( buffer, sizeof(buffer)-1, "%s", str );

Works pretty well. No buffer overrun, in the new standard
rc == strlen(str), in old standard rc == -1, thus I have a way
to check that there was the potential overrun.

The problem arises when strlen(str) > sizeof(buffer).
In this situation the buffer will contain trunkated version
of str, I have the information that the string was
trunkated, but no way to say "hey, I have flushed the
buffer, please continue from the place where you
stopped". There must to be an alternative implementation
of snprintf() to do exactly what I want.

You still haven't explained why you can't just enlarge the buffer
and rerun snprintf in the enlarged buffer.
 
C

CBFalconer

dimka said:
This is funny :) Everyone can cite the standard. But this is EXACTLY
the code that does NOT work.

Don't top-post. Read the code again. Note that it first tests for
overflow, and if such would occur flushes the buffer and marks it
empty. The only assumption is that a single output string fits in
the buffer.

This design is intended to reuse a buffer, but mitigate the system
interruption to perform actual output.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943
 
C

CBFalconer

dimka said:
Here is my situation: I have a char buf[1024]; that is used as a
buffer for a log output.
.... snip ...

Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).
Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved, and that the
buffer is never overwritten. It only involves a few lines of code,
and no tortuous rebuilding of the wheel as you advise. The key is
that snprintf(NULL, 0, ....) returns the output length without any
actual output. Ben has pointed out that some libraries do not
adhere to the standard, but that is a QOI issue.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943
 
W

websnarf

CBFalconer said:
dimka said:
Here is my situation: I have a char buf[1024]; that is used as a
buffer for a log output.
... snip ...

Right, in fact you can't. The arguments are always considered formed
once at compile time. You cannot even process the same va_list more
than once (the standards committee added va_copy to C99, but C99 has
not been widely adopted).
Suppose that using of asprintf(), another buffer and other kinds of
malloc() is not an option. Suppose the buffer must be preprocessed
before it hits the actual io.

Is there any open/closed libraries that solve the problem? Are there
any discussions held on the subject? Please provide the references.

The regulars in this group *cannot* help you. They will cite the
standard, and cannot read well enough to see that you say specifically
that you cannot call malloc() or asprintf().

Ok, the only straight forward "in standard" solution that is possible
for you is to use vfprintf to write the output to a file, then read
back that file in chunks that you can then send to your real output.
Of course this involves the file system and is very likely to be highly
distasteful (what are you supposed to do when the disk is full?).

If you take the trouble to read the solution I posted, you will
find that there are no calls to malloc etc. involved,

I *DID* read your post. It just plain doesn't work, as the OP was
immediately able to point out. But that's just par for the course with
you, so I didn't bother pointing it out directly.
[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.
[...] It only involves a few lines of code,
and no tortuous rebuilding of the wheel as you advise.

And that's fine if the OP doesn't care to have a working solution. My
solution is only meant for the very narrow goal of correctly answering
the OP's question as asked. As to the much broader scope of irrelevant
answers that obviously don't work, I'll leave that to you.
[...] The key is
that snprintf(NULL, 0, ....) returns the output length without any
actual output. Ben has pointed out that some libraries do not
adhere to the standard, but that is a QOI issue.

*MANY* libraries do not adhere to the standard. Go look at the source
for asprintf, or in fact my bformat(a) functions in Bstrlib. We go to
very extreme lengths to tease out the results we want because of the
really poor state of standards compliance on that function.

QOI issue or not, that OP was not asking for a platform specific
solution (isn't that supposed to be OT in this newsgroup?) and was not
asking for a non-functional solution (which you and Ben seemed happy to
provide) either.
 
C

CBFalconer

CBFalconer wrote:
.... snip ...

I *DID* read your post. It just plain doesn't work, as the OP was
immediately able to point out. But that's just par for the course
with you, so I didn't bother pointing it out directly.

No you didn't read the code, or you are incapable of reading
standard C. From your cavalier attitude to standards I suspect the
latter. I even posted a portion of the standard to justify the
technique.
[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.

Here I meant that there never is any overflow. Poorly worded by
me. I guess you require very short monosyllabic phrases. Buffers
are normally intended to be filled and emptied. Believe it or not,
memory is a limited resource.

I apologize for using three and four syllable words, but I find it
hard to express myself without them. You can find most of them in
any dictionary.

--
"The power of the Executive to cast a man into prison without
formulating any charge known to the law, and particularly to
deny him the judgement of his peers, is in the highest degree
odious and is the foundation of all totalitarian government
whether Nazi or Communist." -- W. Churchill, Nov 21, 1943
 
W

websnarf

CBFalconer said:
No you didn't read the code, or you are incapable of reading
standard C. From your cavalier attitude to standards I suspect the
latter. I even posted a portion of the standard to justify the
technique.

*Sigh*. Ok, from the top:

1. templgh = snprintf(NULL, 0, format, whatever);
2. if (templgh + bufindex >= sizeof(buffer) {
3. flushbuf(buffer); bufindex = 0;
4. }
5. bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

Line 1 has a well known problem; what snprintf outputs on different
systems is just inconsistent, so your test on line 2 just does not
correctly identify the incremental overflow case. You can claim this
is just a violation of the standard, so we will ignore your contempt
for real world systems for the moment.

The first argument of Line 5 *might* make sense if you've declared
buffer to be char buffer[<something>][1024] or something like that, but
then we are screwed back on Line 2 with the sizeof(buffer). Ok, so
just looking at the "units" of each parameter, it seems that you
probably meant &buffer[bufindex] -- I don't know how to proceed in
analyzing your code if I can't make this assumption.

Ok, as the OP points out, "whatever" can be a mass of stuff including a
single 8.5 megabyte string. Now if sizeof(buffer) < the length of the
contents of a formatted "whatever", then what the hell would is line 5
going to do? It will just truncate whatever it is you are trying to
sprintf down to sizeof(buffer)-1 characters. So that's just a wrong
result.

Furthermore format and "whatever" are a complex sequence of arguments
-- restarting them, and this sprintf somehow, just isn't going to cut
it; it can't be done. This is the crux of his problem, and this is
what he is asking. He made the point clearer in a follow-up post (I
got it the first time.) Your ignoring this problem is an interesting
way of trying to answer it, but that doesn't help the OP.
[...] and that the buffer is never overwritten.

I don't even know what you mean here -- even *your* solution reuses the
buffer, but that is only step 1 in trying to solve this.

Here I meant that there never is any overflow.

I see. Well as I point out above, that isn't the problem with your
solution anyways.
[...] Poorly worded by
me. I guess you require very short monosyllabic phrases.

Its not poorly worded -- its wrongly worded. I only require correct
sentences using correct words. I'm fine with you simply correcting
yourself after the fact here (I'm hardly immune to typos myself) but of
course, you had to push it.
 
D

dimka

There are few reasons:

The application must use minimum heap memory (application environment
is close
to embeded applications, sory but I cant disclose the exact details).
The malloc() is very
expensive. Also, it is very important to keep low rate of the memory
leaks, while allocating the buffer dynamically will require later
clean-up in all possible exception
handling situations. This will increase chanse of the memory leaks.
 
D

dimka

You are right. The original question was "does anybody know about open
source vpprintf() implementation?". The problem is obvious and exists
since sprintf() was invented.
Thus I hope somebody already solved it.

Anyway, thanks for your answers guys!
 
C

Chris Torek

[someone -- I cannot quite untangle the attributions at this point --
wrote]
bufindex += snprintf(buffer[bufindex],
sizeof(buffer) - 1 - bufindex,
format, whatever);

This code is obviously wrong: the first argument needs to be
&buffer[bufindex] (or equivalent).

Lets assume that the buffer is empty, thus no need to flush it.
then

rc = snprintf( buffer, sizeof(buffer)-1, "%s", str );

Works pretty well. No buffer overrun, in the new standard
rc == strlen(str), in old standard rc == -1, thus I have a way
to check that there was the potential overrun.

The problem arises when strlen(str) > sizeof(buffer).
In this situation the buffer will contain trunkated version
of str, I have the information that the string was
trunkated, but no way to say "hey, I have flushed the
buffer, please continue from the place where you
stopped". There must to be an alternative implementation
of snprintf() to do exactly what I want.

In the general case, this is too hard (MUCH too hard). Consider
what has to happen if, for instance, the format argument is not
just "%s", but instead, we have something like:

int aw, ap; /* width and precision for "a" */
double a; /* value of "a" */
int sw; /* width for "s" */
char *s; /* etc */
int k;
int x;
...
/* there is no need for -1 on sizeof buf; snprintf does that */
x = snprintf(buf, sizeof buf, "%*.*f %*.32s %-12d", aw, ap, a, sw, s, k);

It is possible that, e.g., the printf engine stopped somewhere in
the middle of the "%*.32s", having run out of room for padding or
having run out of the maximum of 32 characters in "s" (or both).
The only reasonable method is to start over, discarding the "unwanted"
characters.

Of course, if your format is always exactly "%s", you do not need
snprintf() at all! The "%s" format just copies a string unchanged,
and the length of the string is simply strlen(str). So given the
simplified version above:

size_t i, part, len;
const char *p;

len = strlen(str);
for (i = len, p = str; i > 0; p += part, i -= part) {
/* find out how much of the string fits */
part = sizeof buffer - 1;
if (part > len)
part = len;

/* put it in the buffer */
memcpy(buffer, p, part);
buffer[part] = 0;

... do something with buffer ...
/* or, do it directly with p[0]..p[part-1], eliminating the memcpy */
}

You only need snprintf() if the format is complicated, in which
case, you usually need the "restart from beginning" behavior.

If you really need something fancier, you can always write your
own (possibly limited) printf-like function(s) in Standard C. Or,
if you are willing to go beyond Standard C, you can:

- use the funopen() I put in the 4.4BSD stdio, or
- use the GNU equivalent of funopen()

to "open" a stdio stream with a user-supplied "buffer writing"
function, and then supply that stream to fprintf(). Of course
using such extensions, while convenient, gives up all kinds of
portability.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,731
Messages
2,569,432
Members
44,832
Latest member
GlennSmall

Latest Threads

Top