redirecting stdout

[jseb]

Hello,


I'd like to bufferise stdout. That's it, when something is sent to
stdout FD, it goes instead to a buffer.

(it's for sending this buffer to another computer with network sockets).

I read that "open_memstren" could do that (posix function).

Here is my test:


///////////// CODE START /////////////////

#include<stdio.h>
#include<malloc.h>
#include<unistd.h>

int main(void)
{
char *buffer; //for open_memstream
size_t size;
FILE *memstream = open_memstream(&buffer, &size);

int stdout_copy = dup(1); //save stdout

stdout = memstream; //redirection

printf("will be put to the buffer");
printf("will be print on stdout!\n");
printf(" another one bufferised");

fclose(memstream);

stdout = fdopen(stdout_copy ,"w+");
printf("stream: %s , size: %d\n", buffer,(int)size);

free(buffer);

return 0;
}

///////////// CODE END /////////////////

It works, except that all strings terminated with "\n" are sent directly
to the terminal !

If you launch this program, you get:


$ ./a.out
will be print on stdout!
stream: will be put to the buffer another one bufferised , size: 48


Do you have any idea about this strange behaviour ?


Thank you.

 

[Lew Pitcher]

Hello,


I'd like to bufferise stdout. That's it, when something is sent to
stdout FD, it goes instead to a buffer.

(it's for sending this buffer to another computer with network sockets).

I read that "open_memstren" could do that (posix function). [snip]
Do you have any idea about this strange behaviour ?

Because open_memstream(), dup(), and fdopen() are not Standard C functions,
I think that you should ask this question in one of the Unix programmer
newsgroups. Perhaps comp.unix.programmer or comp.unix.questions, or even
one of the comp.os.linux.development.* newsgroups would be a proper venue.

<OT>
I don't see anything *obviously* wrong with your code. But, that means
nothing. Have you tried varying the "buffering" using the
setbuf()/setvbuf() functions? There may be some subtle interaction between
the C standard buffering controlled by the setbuf()/setvbuf() calls and the
POSIX buffering initiated by the open_memstream() call that we are not
aware of.
</OT>

HTH

 

[Lew Pitcher]

Hello,


I'd like to bufferise stdout. That's it, when something is sent to
stdout FD, it goes instead to a buffer.

(it's for sending this buffer to another computer with network sockets).

[snip]

Perhaps you are trying to solve the wrong problem?
Instead of rebuffering stdout to memory, so that you can send "this buffer
to another computer with network sockets", why don't you externally pipe
stdout into a sockets-enabled sender?

That is to say, externally,
your_program | nc target.host.name 1234
using "netcat" (nc) or some other tool.

That's gotta be simpler than your programmatic alternatives:
1) continue using stdout-dependant I/O, and redirect to a buffer for later
transmission, or
2) change all I/O to memory buffered I/O (sprintf and the like) to redirect
to a buffer for later transmissiom.

With the use of an external helper, you don't have to change any program
code, and still get your network transmission.

HTH

 

[Les Cargill]

Hello,

Do you have any idea about this strange behaviour ?


Thank you.

It's a bit ambiguous what you mean by 'buffering'.
Unix/Linux has the "tee" tool:

someprog | tee outfile

in which case, you'd not need to make any changes to your program.

 

[jseb]

Hello,

I'm going to put this question in comp.unix.programmer, as you
suggested.

And, alas, i had no success with flushing after each printf.

Instead of rebuffering stdout to memory, so that you can send "this buffer
to another computer with network sockets", why don't you externally pipe
stdout into a sockets-enabled sender?

Yes, piping to a socket would be far simpler…

My environment works like this:

A listening Lua server , which purpose is to execute Lua scripts received.
A terminal connected to the Lua server, which send to it Lua scripts.

The scripts can of course print messages on stdout , so i have to send
them to the client which has sent the commands.

So for piping, i have to know the address of the server, which i don't
know when i launch the server. I could send informations for initiating
the «send» server, but it would need something more complex than netcat
or socat (i use socat at the client side for sending orders).

The problem is the Lua script: it handles stdout only with standard C
functions, so i have to find a solution for «hijacking» its standard
output. The plan is to redirecting stdout to a buffer, sending the
buffer to the caller, and then restoring stdout (i can call easily C
functions from Lua context).

Maybe i'm doing it wrong, but i'm quite sure i cannot solve it with a
pipe.
I had thought of launching the Lua script with a «popen», for getting
all its output easily. I think i will use «popen» if i cannot get off
lightly.

Thank you for reading and taking the time to answer.

 

[Eric Sosman]

Hello,
[...]

What Lew and Les said. One thing you might want to
highlight when you contact other groups is

stdout = memstream; //redirection

.... which reeks of wrongness. See the freopen() function.

 

[Kenny McCormack]

Hello,

I'm going to put this question in comp.unix.programmer, as you
suggested.

(Responding in comp.lang.c - trying to avoid the usual "Oh, you can't talk
about that here" nonsense)

Now, on to substance. I found your post quite interesting. I have, in
fact, faced this problem myself and found and implemented a quite ingenious
(IMHO), albeit entirely non-portable, solution to it. My solution involved
cracking open stdio.h and reverse engineering the FILE structure. As I
said, quite natty, but it is the sort of thing that gives the CLC'ers heart
seizures. Please let me know if you are interested in the details of my
method.

Obviously, the key question here is: How portable do you need to be? Do
you need to be portable at all - or do you just need to "get 'er done"?

The point is, I don't think, based on my reading of the indicated "man"
pages, that it is any more possible to do this "portably" in the POSIX
context than it would be in the "pure ANSI C" context.

P.S. I think that, in at least some implementations, "stdout" (and
similar) are constants - might even be macros - and cannot be assigned to.
(I know I hit this issue recently, trying, IIRC, to do something similar to
what you propose in this thread).

Even if it is not a constant, I don't think there is any guarantee (in C or
in POSIX) that assigning something new to it ("stdout") will have the
desired effect.

HTH

--
(The Republican mind, in a nutshell)
You believe things that are incomprehensible, inconsistent, impossible
because we have commanded you to believe them; go then and do what is
unjust because we command it. Such people show admirable reasoning. Truly,
whoever is able to make you absurd is able to make you unjust. If the
God-given understanding of your mind does not resist a demand to believe
what is impossible, then you will not resist a demand to do wrong to that
God-given sense of justice in your heart. As soon as one faculty of your
soul has been dominated, other faculties will follow as well. And from this
derives all those crimes of religion which have overrun the world.

(Alternative condensed translation)
"Those who can make you believe absurdities, can make you commit atrocities".

 

[jseb]

Please let me know if you are interested in the details of my method.

Of course i'm interested.

As i've had a warm welcome here, i don't want raise blood pressure of
the regular visitors. If it's too off-topic, don't hesitate to write to
my Reply-To address, which is valid.

The portability level i'd like to reach is : «works on Linux with recent
libc, on an amd64 architecture».

Here is the best i can do, based on all the remarks here (still no
posting in comp.unix.programming, as i did some research and reading
instead):

////////// BEGIN CODE /////////

#include<stdio.h>
#include<malloc.h>
#include<unistd.h> //dup, open, close…
#include<string.h>

int main(void)
{

char *buffer; //open_memstream parameter
size_t size; //me too

int stdout_copy = dup(1); //save stdout

fclose(stdout); //stop writing in the term
stdout = open_memstream(&buffer,&size); //redirection

printf("will be put to the buffer");
fflush(stdout);
printf("gaaaahh i'm lost in the 4th dimension!\n");
fflush(stdout);
printf(" and again in the buffer");
fflush(stdout);

fclose(stdout); //close redirected stdio
stdout = fdopen(stdout_copy ,"w"); //restore normal behaviour

printf("stream: %s , size: %d\n", buffer,(int)size);

free(buffer);

return 0;
}

////////// END CODE /////////


Thank again for all the answers.


YIH!
(yes it helped!)

 

[Lew Pitcher]

Of course i'm interested.

As i've had a warm welcome here, i don't want raise blood pressure of
the regular visitors. If it's too off-topic, don't hesitate to write to
my Reply-To address, which is valid.

The portability level i'd like to reach is : «works on Linux with recent
libc, on an amd64 architecture».

Here is the best i can do, based on all the remarks here (still no
posting in comp.unix.programming, as i did some research and reading
instead):

////////// BEGIN CODE /////////

#include<stdio.h>
#include<malloc.h>
#include<unistd.h> //dup, open, close…
#include<string.h>

int main(void)
{

char *buffer; //open_memstream parameter
size_t size; //me too

int stdout_copy = dup(1); //save stdout

fclose(stdout); //stop writing in the term
stdout = open_memstream(&buffer,&size); //redirection

printf("will be put to the buffer");
fflush(stdout);
printf("gaaaahh i'm lost in the 4th dimension!\n");

Remember, printf() returns the number of characters "printed", or (if an
output error is encountered) a negative value.

Your results suggest that the 2nd printf() is having a problem. Have you
confirmed that by checking the return value from the printf() call?

FWIW, I did, and got surprising results.....

I changed the code to...

if (printf("gaaaahh i'm lost in the 4th dimension!\n") < 0)
perror("2nd printf()");

and got, as output
stream: will be put to the buffergaaaahh i'm lost in the 4th dimension!
and again in the buffer , size: 88


Something odd is happening here.

[snip]

 

[Lew Pitcher]

On Tuesday 16 July 2013 19:08, in comp.lang.c,

Remember, printf() returns the number of characters "printed", or (ifan
output error is encountered) a negative value.

Your results suggest that the 2nd printf() is having a problem. Have you
confirmed that by checking the return value from the printf() call?

FWIW, I did, and got surprising results.....

I changed the code to...

if (printf("gaaaahh i'm lost in the 4th dimension!\n") < 0)
perror("2nd printf()");

and got, as output
stream: will be put to the buffergaaaahh i'm lost in the 4th dimension!
and again in the buffer , size: 88


Something odd is happening here.

FWIW, GCC version 4.2.4 optimizes that second printf() into a puts() call
unless you check the return value, or otherwise give it some reason notto.

Perhaps your compiler is optimizing the printf() call in such a way as to
avoid the memstream buffering?

 

[Keith Thompson]

////////// BEGIN CODE /////////

#include<stdio.h>
#include<malloc.h>
#include<unistd.h> //dup, open, close…
#include<string.h>

int main(void)
{

char *buffer; //open_memstream parameter
size_t size; //me too

int stdout_copy = dup(1); //save stdout

fclose(stdout); //stop writing in the term
stdout = open_memstream(&buffer,&size); //redirection

Assuming that the above works as intended (I'm skeptical about just
assigning a value to stdout) ...

printf("will be put to the buffer");
fflush(stdout);
printf("gaaaahh i'm lost in the 4th dimension!\n");
fflush(stdout);
printf(" and again in the buffer");
fflush(stdout);

fclose(stdout); //close redirected stdio

The array pointed to by `buffer` should now contain something like

"will be put to the buffergaaaahh i'm lost in the 4th dimension!\n and again in the buffer"

but *without* a terminating '\0' character.

stdout = fdopen(stdout_copy ,"w"); //restore normal behaviour

printf("stream: %s , size: %d\n", buffer,(int)size);

And now you pass `buffer` to `printf`, with a format string that assumes
it points to a properly '\0'-terminated string.

I suggest either printing a '\0' to the buffer before fclose'ing it or
printing the buffer's contents in a way that doesn't assume it contains
a string; the latter probably makes more sense.

(Incidentally, I suspect the fflush() calls are unnecessary; even if the
printf() calls don't immediately update the buffer, the fclose() call
should.

 

[Lew Pitcher]

Assuming that the above works as intended (I'm skeptical about just
assigning a value to stdout) ...


The array pointed to by `buffer` should now contain something like

"will be put to the buffergaaaahh i'm lost in the 4th dimension!\n and
again in the buffer"

but *without* a terminating '\0' character.


And now you pass `buffer` to `printf`, with a format string that assumes
it points to a properly '\0'-terminated string.

I suggest either printing a '\0' to the buffer before fclose'ing it or
printing the buffer's contents in a way that doesn't assume it contains
a string; the latter probably makes more sense.

Wouldn't

printf("stream: %*s , size: %d\n", (int)size, buffer, (int)size);

be suitable here?


[snip]

 

[Lew Pitcher]

On Tuesday 16 July 2013 19:27, in comp.lang.c,

On Tuesday 16 July 2013 19:08, in comp.lang.c,


FWIW, GCC version 4.2.4 optimizes that second printf() into a puts() call
unless you check the return value, or otherwise give it some reason not
to.

Perhaps your compiler is optimizing the printf() call in such a way as to
avoid the memstream buffering?

I've played around a bit with the OP's code, and (barring any UB introduced
by the nonstandard function calls) suspect that his problem is related to
his implementation of the puts() function of the standard C library.

I've found that
- in the misbehaving version of the original code, gcc optimizes the
original printf("...\n"); into a puts("...")
- in any version that forces optimization to leave the original printf()
alone, the aberrant behaviour goes away, and the code generates the
expected results
- explicitly substituting a puts("...") in the code, in place of the
original printf("...\n") reproduces the abberant behaviour
- explicitly substituting a printf("...%s...\n","4th") in the code, in
place of the original printf("...4th...\n") causes the abberant behaviour
to go away, and the expected results to appear
- explicitly using fputs("...\n",stdout) in the code, in place of the
original printf("...\n") causes the abberant behaviour to go away, and
the expected results to appear

In other words, it appears that certain optimizations performed by the
compiler cause puts() to be substituted for printf(), and that puts() does
not direct it's output to the buffer provided by the open_memstream() call.
And, that sounds to me like a bug in the glibc implementation of puts()..

 

[Keith Thompson]

Wouldn't

printf("stream: %*s , size: %d\n", (int)size, buffer, (int)size);

be suitable here?

I believe it would. I wasn't sure until I checked the standard, but the
definition for "%s" is carefully written to permit a non-terminated
character array as long as a precision is specified.

Calling fwrite() might be a good alternative, less likely to send a
typical reader to the standard to understand the code:

printf("stream: ");
fwrite(buffer, 1, size, stdout);
printf(" , size: %zu\n", size);

 

[James Kuyper]

Assuming that the above works as intended (I'm skeptical about just
assigning a value to stdout) ...

The standard says that stderr, stdin, and stdout "are expressions of
type ‘‘pointer to FILE’’ that point to the FILE objects associated,
respectively, with the standard error, input, and output streams."
(7.21.1p3)
"An assignment operator shall have a modifiable lvalue as its left
operand." (6.5.16p2)

<stdio.h> could contain:
extern FILE __standard_streams[];
#define stdout (__standard_streams+2)

I believe that something like this is, in fact, not uncommon.

 

[Kenny McCormack]

Assuming that the above works as intended (I'm skeptical about just
assigning a value to stdout) ...

The standard says that stderr, stdin, and stdout "are expressions of
type ‘‘pointer to FILE’’ that point to the FILE objects associated,
respectively, with the standard error, input, and output streams."
(7.21.1p3)
"An assignment operator shall have a modifiable lvalue as its left
operand." (6.5.16p2)

<stdio.h> could contain:
extern FILE __standard_streams[];
#define stdout (__standard_streams+2)

I believe that something like this is, in fact, not uncommon.

But do keep in mind that if that were the problem (in the instant cast),
you'd get a compile error (for trying to assign to a non-lvalue).

So, your comment (although valid) is probably not relevant.

 

[Tim Rentsch]

stdout = open_memstream(&buffer,&size); //redirection

Assuming that the above works as intended (I'm skeptical about
just assigning a value to stdout) ...

The standard says that stderr, stdin, and stdout "are expressions
of type ''pointer to FILE'' that point to the FILE objects
associated, respectively, with the standard error, input, and
output streams." (7.21.1p3)
"An assignment operator shall have a modifiable lvalue as its
left operand." (6.5.16p2)

<stdio.h> could contain:
extern FILE __standard_streams[];
#define stdout (__standard_streams+2)

I believe that something like this is, in fact, not uncommon.

But do keep in mind that if that were the problem (in the
instant cast), you'd get a compile error (for trying to assign
to a non-lvalue).

So, your comment (although valid) is probably not relevant.

It is relevant to Keith Thompson's comment, which is what
is being responded to.

 

[Tim Rentsch]

Wouldn't

printf("stream: %*s , size: %d\n", (int)size, buffer, (int)size);

be suitable here?

You mean

printf("stream: %.*s , size: %d\n", (int)size, buffer, (int)size);

(notice the . before the *).

 

[Lew Pitcher]

You mean

printf("stream: %.*s , size: %d\n", (int)size, buffer, (int)size);

(notice the . before the *).

Oh, yes. I do. Or, at least, that's what I was looking for.

Thanks for the pointer. I learn something new with each discussion here.

 

[Tim Rentsch]

[snip]

My environment works like this:

A listening Lua server , which purpose is to execute Lua scripts
received. A terminal connected to the Lua server, which send to
it Lua scripts.

The scripts can of course print messages on stdout, so i have to
send them to the client which has sent the commands.

So for piping, i have to know the address of the server, which i
don't know when i launch the server. I could send informations
for initiating the <<send>> server, but it would need something
more complex than netcat or socat (i use socat at the client side
for sending orders).

The problem is the Lua script: it handles stdout only with
standard C functions, so i have to find a solution for
<<hijacking>> its standard output. The plan is to redirecting
stdout to a buffer, sending the buffer to the caller, and then
restoring stdout (i can call easily C functions from Lua
context).

Maybe i'm doing it wrong, but i'm quite sure i cannot solve it
with a pipe.

I had thought of launching the Lua script with a <<popen>>, for
getting all its output easily. I think i will use <<popen>> if i
cannot get off lightly.

Thank you for reading and taking the time to answer.

It looks like popen() does what you want, will be easy to use,
and easy to make work reliably. Given that, why are fiddling
around with changing stdout? I know the up front costs of
the stdout approach look like they might be lower than using
popen(), but the potential downstream costs (and downstream
hurt) associated with the stdout route are much higher than
just using popen() from the beginning.