redirecting stdout

Discussion in 'C Programming' started by jseb, Jul 16, 2013.

  1. jseb

    jseb Guest

    Hello,


    I'd like to bufferise stdout. That's it, when something is sent to
    stdout FD, it goes instead to a buffer.

    (it's for sending this buffer to another computer with network sockets).

    I read that "open_memstren" could do that (posix function).

    Here is my test:


    ///////////// CODE START /////////////////

    #include<stdio.h>
    #include<malloc.h>
    #include<unistd.h>

    int main(void)
    {
    char *buffer; //for open_memstream
    size_t size;
    FILE *memstream = open_memstream(&buffer, &size);

    int stdout_copy = dup(1); //save stdout

    stdout = memstream; //redirection

    printf("will be put to the buffer");
    printf("will be print on stdout!\n");
    printf(" another one bufferised");

    fclose(memstream);

    stdout = fdopen(stdout_copy ,"w+");
    printf("stream: %s , size: %d\n", buffer,(int)size);

    free(buffer);

    return 0;
    }

    ///////////// CODE END /////////////////

    It works, except that all strings terminated with "\n" are sent directly
    to the terminal !

    If you launch this program, you get:


    $ ./a.out
    will be print on stdout!
    stream: will be put to the buffer another one bufferised , size: 48


    Do you have any idea about this strange behaviour ?


    Thank you.
     
    jseb, Jul 16, 2013
    #1
    1. Advertisements

  2. jseb

    Lew Pitcher Guest

    Because open_memstream(), dup(), and fdopen() are not Standard C functions,
    I think that you should ask this question in one of the Unix programmer
    newsgroups. Perhaps comp.unix.programmer or comp.unix.questions, or even
    one of the comp.os.linux.development.* newsgroups would be a proper venue.

    <OT>
    I don't see anything *obviously* wrong with your code. But, that means
    nothing. Have you tried varying the "buffering" using the
    setbuf()/setvbuf() functions? There may be some subtle interaction between
    the C standard buffering controlled by the setbuf()/setvbuf() calls and the
    POSIX buffering initiated by the open_memstream() call that we are not
    aware of.
    </OT>

    HTH
     
    Lew Pitcher, Jul 16, 2013
    #2
    1. Advertisements

  3. jseb

    Lew Pitcher Guest

    [snip]

    Perhaps you are trying to solve the wrong problem?
    Instead of rebuffering stdout to memory, so that you can send "this buffer
    to another computer with network sockets", why don't you externally pipe
    stdout into a sockets-enabled sender?

    That is to say, externally,
    your_program | nc target.host.name 1234
    using "netcat" (nc) or some other tool.

    That's gotta be simpler than your programmatic alternatives:
    1) continue using stdout-dependant I/O, and redirect to a buffer for later
    transmission, or
    2) change all I/O to memory buffered I/O (sprintf and the like) to redirect
    to a buffer for later transmissiom.

    With the use of an external helper, you don't have to change any program
    code, and still get your network transmission.

    HTH
     
    Lew Pitcher, Jul 16, 2013
    #3
  4. jseb

    Les Cargill Guest

    It's a bit ambiguous what you mean by 'buffering'.
    Unix/Linux has the "tee" tool:

    someprog | tee outfile

    in which case, you'd not need to make any changes to your program.
     
    Les Cargill, Jul 16, 2013
    #4
  5. jseb

    jseb Guest

    Hello,

    I'm going to put this question in comp.unix.programmer, as you
    suggested.

    And, alas, i had no success with flushing after each printf.

    Yes, piping to a socket would be far simpler…

    My environment works like this:

    A listening Lua server , which purpose is to execute Lua scripts received.
    A terminal connected to the Lua server, which send to it Lua scripts.

    The scripts can of course print messages on stdout , so i have to send
    them to the client which has sent the commands.

    So for piping, i have to know the address of the server, which i don't
    know when i launch the server. I could send informations for initiating
    the «send» server, but it would need something more complex than netcat
    or socat (i use socat at the client side for sending orders).

    The problem is the Lua script: it handles stdout only with standard C
    functions, so i have to find a solution for «hijacking» its standard
    output. The plan is to redirecting stdout to a buffer, sending the
    buffer to the caller, and then restoring stdout (i can call easily C
    functions from Lua context).

    Maybe i'm doing it wrong, but i'm quite sure i cannot solve it with a
    pipe.
    I had thought of launching the Lua script with a «popen», for getting
    all its output easily. I think i will use «popen» if i cannot get off
    lightly.

    Thank you for reading and taking the time to answer.
     
    jseb, Jul 16, 2013
    #5
  6. jseb

    Eric Sosman Guest

    What Lew and Les said. One thing you might want to
    highlight when you contact other groups is
    .... which reeks of wrongness. See the freopen() function.
     
    Eric Sosman, Jul 16, 2013
    #6
  7. (Responding in comp.lang.c - trying to avoid the usual "Oh, you can't talk
    about that here" nonsense)

    Now, on to substance. I found your post quite interesting. I have, in
    fact, faced this problem myself and found and implemented a quite ingenious
    (IMHO), albeit entirely non-portable, solution to it. My solution involved
    cracking open stdio.h and reverse engineering the FILE structure. As I
    said, quite natty, but it is the sort of thing that gives the CLC'ers heart
    seizures. Please let me know if you are interested in the details of my
    method.

    Obviously, the key question here is: How portable do you need to be? Do
    you need to be portable at all - or do you just need to "get 'er done"?

    The point is, I don't think, based on my reading of the indicated "man"
    pages, that it is any more possible to do this "portably" in the POSIX
    context than it would be in the "pure ANSI C" context.

    P.S. I think that, in at least some implementations, "stdout" (and
    similar) are constants - might even be macros - and cannot be assigned to.
    (I know I hit this issue recently, trying, IIRC, to do something similar to
    what you propose in this thread).

    Even if it is not a constant, I don't think there is any guarantee (in C or
    in POSIX) that assigning something new to it ("stdout") will have the
    desired effect.

    HTH

    --
    (The Republican mind, in a nutshell)
    You believe things that are incomprehensible, inconsistent, impossible
    because we have commanded you to believe them; go then and do what is
    unjust because we command it. Such people show admirable reasoning. Truly,
    whoever is able to make you absurd is able to make you unjust. If the
    God-given understanding of your mind does not resist a demand to believe
    what is impossible, then you will not resist a demand to do wrong to that
    God-given sense of justice in your heart. As soon as one faculty of your
    soul has been dominated, other faculties will follow as well. And from this
    derives all those crimes of religion which have overrun the world.

    (Alternative condensed translation)
    "Those who can make you believe absurdities, can make you commit atrocities".
     
    Kenny McCormack, Jul 16, 2013
    #7
  8. jseb

    jseb Guest

    Of course i'm interested.

    As i've had a warm welcome here, i don't want raise blood pressure of
    the regular visitors. If it's too off-topic, don't hesitate to write to
    my Reply-To address, which is valid.

    The portability level i'd like to reach is : «works on Linux with recent
    libc, on an amd64 architecture».

    Here is the best i can do, based on all the remarks here (still no
    posting in comp.unix.programming, as i did some research and reading
    instead):

    ////////// BEGIN CODE /////////

    #include<stdio.h>
    #include<malloc.h>
    #include<unistd.h> //dup, open, close…
    #include<string.h>

    int main(void)
    {

    char *buffer; //open_memstream parameter
    size_t size; //me too

    int stdout_copy = dup(1); //save stdout

    fclose(stdout); //stop writing in the term
    stdout = open_memstream(&buffer,&size); //redirection

    printf("will be put to the buffer");
    fflush(stdout);
    printf("gaaaahh i'm lost in the 4th dimension!\n");
    fflush(stdout);
    printf(" and again in the buffer");
    fflush(stdout);

    fclose(stdout); //close redirected stdio
    stdout = fdopen(stdout_copy ,"w"); //restore normal behaviour

    printf("stream: %s , size: %d\n", buffer,(int)size);

    free(buffer);

    return 0;
    }

    ////////// END CODE /////////


    Thank again for all the answers.


    YIH!
    (yes it helped!)
     
    jseb, Jul 16, 2013
    #8
  9. jseb

    Lew Pitcher Guest

    Remember, printf() returns the number of characters "printed", or (if an
    output error is encountered) a negative value.

    Your results suggest that the 2nd printf() is having a problem. Have you
    confirmed that by checking the return value from the printf() call?

    FWIW, I did, and got surprising results.....

    I changed the code to...

    if (printf("gaaaahh i'm lost in the 4th dimension!\n") < 0)
    perror("2nd printf()");

    and got, as output
    stream: will be put to the buffergaaaahh i'm lost in the 4th dimension!
    and again in the buffer , size: 88


    Something odd is happening here.

    [snip]
     
    Lew Pitcher, Jul 17, 2013
    #9
  10. jseb

    Lew Pitcher Guest

    On Tuesday 16 July 2013 19:08, in comp.lang.c,
    FWIW, GCC version 4.2.4 optimizes that second printf() into a puts() call
    unless you check the return value, or otherwise give it some reason notto.

    Perhaps your compiler is optimizing the printf() call in such a way as to
    avoid the memstream buffering?
     
    Lew Pitcher, Jul 17, 2013
    #10
  11. Assuming that the above works as intended (I'm skeptical about just
    assigning a value to stdout) ...
    The array pointed to by `buffer` should now contain something like

    "will be put to the buffergaaaahh i'm lost in the 4th dimension!\n and again in the buffer"

    but *without* a terminating '\0' character.
    And now you pass `buffer` to `printf`, with a format string that assumes
    it points to a properly '\0'-terminated string.

    I suggest either printing a '\0' to the buffer before fclose'ing it or
    printing the buffer's contents in a way that doesn't assume it contains
    a string; the latter probably makes more sense.

    (Incidentally, I suspect the fflush() calls are unnecessary; even if the
    printf() calls don't immediately update the buffer, the fclose() call
    should.
     
    Keith Thompson, Jul 17, 2013
    #11
  12. jseb

    Lew Pitcher Guest

    Wouldn't

    printf("stream: %*s , size: %d\n", (int)size, buffer, (int)size);

    be suitable here?


    [snip]
     
    Lew Pitcher, Jul 17, 2013
    #12
  13. jseb

    Lew Pitcher Guest

    On Tuesday 16 July 2013 19:27, in comp.lang.c,
    I've played around a bit with the OP's code, and (barring any UB introduced
    by the nonstandard function calls) suspect that his problem is related to
    his implementation of the puts() function of the standard C library.

    I've found that
    - in the misbehaving version of the original code, gcc optimizes the
    original printf("...\n"); into a puts("...")
    - in any version that forces optimization to leave the original printf()
    alone, the aberrant behaviour goes away, and the code generates the
    expected results
    - explicitly substituting a puts("...") in the code, in place of the
    original printf("...\n") reproduces the abberant behaviour
    - explicitly substituting a printf("...%s...\n","4th") in the code, in
    place of the original printf("...4th...\n") causes the abberant behaviour
    to go away, and the expected results to appear
    - explicitly using fputs("...\n",stdout) in the code, in place of the
    original printf("...\n") causes the abberant behaviour to go away, and
    the expected results to appear

    In other words, it appears that certain optimizations performed by the
    compiler cause puts() to be substituted for printf(), and that puts() does
    not direct it's output to the buffer provided by the open_memstream() call.
    And, that sounds to me like a bug in the glibc implementation of puts()..
     
    Lew Pitcher, Jul 17, 2013
    #13
  14. I believe it would. I wasn't sure until I checked the standard, but the
    definition for "%s" is carefully written to permit a non-terminated
    character array as long as a precision is specified.

    Calling fwrite() might be a good alternative, less likely to send a
    typical reader to the standard to understand the code:

    printf("stream: ");
    fwrite(buffer, 1, size, stdout);
    printf(" , size: %zu\n", size);
     
    Keith Thompson, Jul 17, 2013
    #14
  15. jseb

    James Kuyper Guest

    The standard says that stderr, stdin, and stdout "are expressions of
    type ‘‘pointer to FILE’’ that point to the FILE objects associated,
    respectively, with the standard error, input, and output streams."
    (7.21.1p3)
    "An assignment operator shall have a modifiable lvalue as its left
    operand." (6.5.16p2)

    <stdio.h> could contain:
    extern FILE __standard_streams[];
    #define stdout (__standard_streams+2)

    I believe that something like this is, in fact, not uncommon.
     
    James Kuyper, Jul 17, 2013
    #15
  16. But do keep in mind that if that were the problem (in the instant cast),
    you'd get a compile error (for trying to assign to a non-lvalue).

    So, your comment (although valid) is probably not relevant.
     
    Kenny McCormack, Jul 17, 2013
    #16
  17. jseb

    Tim Rentsch Guest

    It is relevant to Keith Thompson's comment, which is what
    is being responded to.
     
    Tim Rentsch, Jul 17, 2013
    #17
  18. jseb

    Tim Rentsch Guest

    You mean

    printf("stream: %.*s , size: %d\n", (int)size, buffer, (int)size);

    (notice the . before the *).
     
    Tim Rentsch, Jul 17, 2013
    #18
  19. jseb

    Lew Pitcher Guest

    Oh, yes. I do. Or, at least, that's what I was looking for.

    Thanks for the pointer. I learn something new with each discussion here.
     
    Lew Pitcher, Jul 17, 2013
    #19
  20. jseb

    Tim Rentsch Guest

    It looks like popen() does what you want, will be easy to use,
    and easy to make work reliably. Given that, why are fiddling
    around with changing stdout? I know the up front costs of
    the stdout approach look like they might be lower than using
    popen(), but the potential downstream costs (and downstream
    hurt) associated with the stdout route are much higher than
    just using popen() from the beginning.
     
    Tim Rentsch, Jul 17, 2013
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.