Any way to capture stdcout output to memory?

Discussion in 'C Programming' started by Jim Langston, May 1, 2008.

  1. Jim Langston

    Jim Langston Guest

    I had asked this in comp.lang.c++ with out any answers that would actually
    work, so I'm hoping someone here may know a way.

    I am calling C library functions that want to output to stdout. I need to
    capture this to memory for use internally inside the program without
    modifying the library calls that actually do the output.

    This is for the GraphViz libraries and it is the output of dot that I'm
    actually trying to capture. I have compiled all the libraries and the dot
    exectuable which outputs to stdcout. There is also an option to output to a
    file and if someone knows a way of redirecting a FILE* to memory that would
    work too. There is an option to output to memory but it is a static buffer
    that is WAY too small and is broken anyway. When trying this way I get a
    memory error and tracing through the code somewhere deep inside the library
    it's attempting to write to a FILE*

    If no one knows of a way I will have to write a libary function to do this
    and have to delve into the GraphViz libraries and try not to come across the
    same bug that the pervious person who was attempting to write to memory
    does.

    One thing I will do in my attempt is not allocate the memory inside the
    library as is currently attempted, but pass in a function pointer to
    allocate the memory (that will probably just wrap malloc and/or remalloc).

    Any help is appreciated. The GraphViz library is pure C.

    --
    Jim Langston
     
    Jim Langston, May 1, 2008
    #1
    1. Advertising

  2. Jim Langston

    Jim Langston Guest

    Richard Heathfield wrote:
    > Jim Langston said:
    >
    >> I had asked this in comp.lang.c++ with out any answers that would
    >> actually
    >> work, so I'm hoping someone here may know a way.
    >>
    >> I am calling C library functions that want to output to stdout. I
    >> need to capture this to memory for use internally inside the program
    >> without modifying the library calls that actually do the output.

    >
    > Surely this sounds like a job for a pipe? Not a standard C concept,
    > granted, but I'll bet you that comp.unix.programmer will know all
    > about it.


    I know about pipes from the OS point of view. The thing is I only one one
    process running. My program will be calling the library functions a lot as
    it produces it's own data to be worked on, then needs to work on the results
    from the library calls that dot uses.

    --
    Jim Langston
     
    Jim Langston, May 1, 2008
    #2
    1. Advertising

  3. Jim Langston

    Jim Langston Guest

    Richard Heathfield wrote:
    > Jim Langston said:
    >
    >> Richard Heathfield wrote:
    >>> Jim Langston said:
    >>>

    > <snip>
    >>>>
    >>>> I am calling C library functions that want to output to stdout. I
    >>>> need to capture this to memory for use internally inside the
    >>>> program without modifying the library calls that actually do the
    >>>> output.
    >>>
    >>> Surely this sounds like a job for a pipe? Not a standard C concept,
    >>> granted, but I'll bet you that comp.unix.programmer will know all
    >>> about it.

    >>
    >> I know about pipes from the OS point of view. The thing is I only
    >> one

    >
    > [want?]
    >
    >> one process running. My program will be calling the library
    >> functions a lot
    >> as it produces it's own data to be worked on, then needs to work on
    >> the results from the library calls that dot uses.

    >
    > Then I think you're going to have to resort to some lib-hackery.
    > Sorry.


    I was actually playing around with createpipe and such and trying to get the
    code to work when I finally realized I was trying to redirect stdin to
    stdout and stdout to stdin. Heh, that would be an endless loop since the
    functoin reads from stdin.

    I think I might have figured out a way, something like:

    functiontocapturestdout();
    functionthatwritestostdout();
    functionthtaclosescapture();

    functiontocapturestdout() would actually have to create a thread and have
    stdout redirected to that thread. That thread would then read from the
    redirected stdout and throw the data into a buffer.
    functiontoclosecapture() would end the process.

    This seems doable, but now seems OT for this newsgroup and I will start
    discuessing the possibilty in microsoft.language.vc.language.

    Thank you for leading me into the right direction.


    --
    Jim Langston
     
    Jim Langston, May 1, 2008
    #3
  4. Jim Langston

    Willem Guest

    Jim wrote:
    ) I know about pipes from the OS point of view. The thing is I only one one
    ) process running. My program will be calling the library functions a lot as
    ) it produces it's own data to be worked on, then needs to work on the results
    ) from the library calls that dot uses.

    You can use a pipe from a single process, but only if that pipe buffers
    enough data so that your library call won't block on the write to stdout.
    If the lib only produces a line at a time that may be possible. Read
    your platform's documentation on pipe() and dup2().

    You might also conceivably be able to link the 'write' call in the library
    to your own function, but that is very platform specific deep black magic.

    Otherwise, your best bet is pointing stdout to a file and then reading
    from it.


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
     
    Willem, May 1, 2008
    #4
  5. Jim Langston

    Jim Langston Guest

    Willem wrote:
    > Jim wrote:
    > ) I know about pipes from the OS point of view. The thing is I only
    > one one ) process running. My program will be calling the library
    > functions a lot as ) it produces it's own data to be worked on, then
    > needs to work on the results ) from the library calls that dot uses.
    >
    > You can use a pipe from a single process, but only if that pipe
    > buffers enough data so that your library call won't block on the
    > write to stdout.
    > If the lib only produces a line at a time that may be possible.


    That's not going to be possible. On a small sample set of data 215 lines
    were produced.

    > Read your platform's documentation on pipe() and dup2().
    >
    > You might also conceivably be able to link the 'write' call in the
    > library
    > to your own function, but that is very platform specific deep black
    > magic.


    Yeah, very deep black magic. Trying to get something not quite that deep,
    but I will if I have too.

    > Otherwise, your best bet is pointing stdout to a file and then reading
    > from it.


    Unfortunately, I need this to be rather fast and file IO being a major
    bottleneck isn't going to work for me. I think I may be able to get away
    with running the call itself it it's own process allowing me to redirect the
    stdout. Not sure yet.

    Thanks.

    --
    Jim Langston
     
    Jim Langston, May 1, 2008
    #5
  6. Jim Langston

    Willem Guest

    Jim wrote:
    ) Willem wrote:
    )> You can use a pipe from a single process, but only if that pipe
    )> buffers enough data so that your library call won't block on the
    )> write to stdout.
    )> If the lib only produces a line at a time that may be possible.
    )
    ) That's not going to be possible. On a small sample set of data 215 lines
    ) were produced.

    I'm really not sure how large the buffer is on a pipe on your platform.
    It may be in the order of 4kb or it could be a whole lot bigger.
    Perhaps you can even tune it to a user-defined size.

    Downside, of course, is that the whole app blocks when you fill the
    buffer. Or make it non-blocking, in which case you get an error.

    )> Otherwise, your best bet is pointing stdout to a file and then reading
    )> from it.
    )
    ) Unfortunately, I need this to be rather fast and file IO being a major
    ) bottleneck isn't going to work for me. I think I may be able to get away
    ) with running the call itself it it's own process allowing me to redirect the
    ) stdout. Not sure yet.

    Well, if you can do that, that would be easiest.
    Note that most of the code will be the same as the single-process
    pipe()-dup2() method, so you can try that first.

    Details on how to do it are platform-specific though so you should ask on
    the newsgroup dedicated to that platform. comp.unix.programmer I guess ?


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
     
    Willem, May 1, 2008
    #6
  7. Jim Langston wrote:
    >
    > Richard Heathfield wrote:
    > > Jim Langston said:

    [...]
    > >> I am calling C library functions that want to output to stdout. I
    > >> need to capture this to memory for use internally inside the program
    > >> without modifying the library calls that actually do the output.

    > >
    > > Surely this sounds like a job for a pipe? Not a standard C concept,
    > > granted, but I'll bet you that comp.unix.programmer will know all
    > > about it.

    >
    > I know about pipes from the OS point of view. The thing is I only one one
    > process running. My program will be calling the library functions a lot as
    > it produces it's own data to be worked on, then needs to work on the results
    > from the library calls that dot uses.


    Then you'll need to resort to some further OT-for-clc method, such
    as connecting stdout to a pipe, and creating a thread to read the
    other side of the pipe.

    How to do this is very OT here, but may be topical for something like
    comp.unix.programmer.

    --
    +-------------------------+--------------------+-----------------------+
    | Kenneth J. Brody | www.hvcomputer.com | #include |
    | kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
    +-------------------------+--------------------+-----------------------+
    Don't e-mail me at: <mailto:>
     
    Kenneth Brody, May 1, 2008
    #7
  8. Jim Langston wrote:
    > Richard Heathfield wrote:
    >> Jim Langston said:
    >>
    >>> I had asked this in comp.lang.c++ with out any answers that would
    >>> actually
    >>> work, so I'm hoping someone here may know a way.
    >>>
    >>> I am calling C library functions that want to output to stdout. I
    >>> need to capture this to memory for use internally inside the program
    >>> without modifying the library calls that actually do the output.

    >> Surely this sounds like a job for a pipe? Not a standard C concept,
    >> granted, but I'll bet you that comp.unix.programmer will know all
    >> about it.

    >
    > I know about pipes from the OS point of view. The thing is I only one one
    > process running. My program will be calling the library functions a lot as
    > it produces it's own data to be worked on, then needs to work on the results
    > from the library calls that dot uses.
    >

    Not standard C, but you can diddle the file descriptors using dup2(),
    saving the old FD and switching it back when you are done.
    --
    clvrmnky <mailto:>

    Direct replies to this address will be blacklisted. Replace "spamtrap"
    with my name to contact me directly.
     
    Clever Monkey, May 1, 2008
    #8
  9. Jim Langston

    Jim Langston Guest

    CBFalconer wrote:
    > Jim Langston wrote:
    >>
    >> I had asked this in comp.lang.c++ with out any answers that would
    >> actually work, so I'm hoping someone here may know a way.
    >>
    >> I am calling C library functions that want to output to stdout.
    >> I need to capture this to memory for use internally inside the
    >> program without modifying the library calls that actually do the
    >> output.

    >
    > All the functions that output to stdout do so as if the final
    > output is implemented by "putc(i, stdout)". So all you have to do
    > is write replacement functions, such as, say, "dputs(char *)". It
    > will call both putc and your own putcm. putcm stuffs the char copy
    > in the memory buffer. Then modify your source to call the new
    > functions in place of the old.


    ".. modify your source..."

    If I was able to modify the source I'd have it write to a memory buffer in
    the first place. The problem is the actual output is being produced deep
    inside a library call. This right now is not an option, although I may have
    to make it an option if I can't find a good way to do it. I think one of
    the methods of redirecting stdout will work, although it looks like I'll
    have to use a thread which I don't like but *shrug*

    --
    Jim Langston
     
    Jim Langston, May 2, 2008
    #9
  10. In article <raDSj.3$>,
    Jim Langston <> wrote:
    ....
    >".. modify your source..."
    >
    >If I was able to modify the source I'd have it write to a memory buffer in
    >the first place. The problem is the actual output is being produced deep
    >inside a library call. This right now is not an option, although I may have
    >to make it an option if I can't find a good way to do it. I think one of
    >the methods of redirecting stdout will work, although it looks like I'll
    >have to use a thread which I don't like but *shrug*


    I, too, have hit this issue. First comment: You (Jim) have made it
    clear that the platforms is Windows, not Unix, right? I think some of
    the posters are assuming Unix. Unix would simplify things, since these
    sorts of things are always easier there. Things like dup(), etc,
    although often implemented in Windows, often don't work quite the same
    under Windows as under Unix.

    On Windows, I was able to solve the problem (very OT for clc, of course)
    by diddling with the elements of the the FILE structure. The basic
    trick is that you set the buffersize very high, so that it _never_ gets
    written out. Then, your main program extracts the information out of
    the stdout buffer (pointed to by one of the elements in the FILE
    structure). I did this using the MinGW compiler - but I would imagine
    it is doable with most compilers.

    I think that the thread solution is actually better, but is, as you say,
    complicated to implement. I wish somebody would do it, though, and make
    the code publicly available.

    P.S. Note that even if source is available and that modifying it to
    call different functions was an option, it is possible that doing so
    would be too difficult - if, for example, the existing code makes many,
    many calls to stdout, intermixing different functions, etc.
     
    Kenny McCormack, May 2, 2008
    #10
  11. In article <fvf269$v80$>,
    Kenny McCormack <> wrote:

    >On Windows, I was able to solve the problem (very OT for clc, of course)
    >by diddling with the elements of the the FILE structure. The basic
    >trick is that you set the buffersize very high, so that it _never_ gets
    >written out.


    If you're willing to do that, there's a much more elegant solution for
    some stdio implementations (I don't know about Windows). These
    implementations store pointers to the underlying i/o functions in the
    FILE struct, so by modifying them you can arrange for your own
    function to be called when the data is flushed. Typically there will
    be some kind of cookie in which you could store, say, a buffer to
    which the data would be copied.

    For example, MacOS-X's FILE structure (which I think it inherits
    from FreeBSD) includes:

    /* operations */
    void *_cookie; /* cookie passed to io functions */
    int (*_close)(void *);
    int (*_read) (void *, char *, int);
    fpos_t (*_seek) (void *, fpos_t, int);
    int (*_write)(void *, const char *, int);

    I would be nice if this was standardised, since it's something that
    people have been wanting to do for decades.

    -- Richard
    --
    :wq
     
    Richard Tobin, May 2, 2008
    #11
  12. Jim Langston

    Chris Torek Guest

    In article <fvft86$21lv$>
    Richard Tobin <> wrote:
    >... there's a much more elegant solution for
    >some stdio implementations (I don't know about Windows). These
    >implementations store pointers to the underlying i/o functions in the
    >FILE struct, so by modifying them you can arrange for your own
    >function to be called when the data is flushed.


    But these are not exposed to users, for various reasons; you
    are only supposed to know about funopen() (and several shorter
    names for specific cases; funopen() is the general case).

    >For example, MacOS-X's FILE structure (which I think it inherits
    >from FreeBSD) includes:
    >
    > /* operations */
    > void *_cookie; /* cookie passed to io functions */
    > int (*_close)(void *);
    > int (*_read) (void *, char *, int);
    > fpos_t (*_seek) (void *, fpos_t, int);
    > int (*_write)(void *, const char *, int);


    This came from my stdio by way of 4.4BSD and then (presumably)
    FreeBSD, yes.

    >I would be nice if this was standardised, since it's something that
    >people have been wanting to do for decades.


    This lets you do:

    struct whatever some_data_struct;
    int read_op(void *, char *, int);
    int write_op(void *, const char *, int);

    ... set up "some_data_struct" as needed ...
    fp = funopen(&some_data_struct, read_op, write_op, NULL, NULL);
    ... handle failure ...
    some_library_function(fp); /* uses fprintf() and/or fgets() */
    ... etc ...

    so that you can capture the output of "well-behaved" library
    functions that write to a stdio "FILE *", feed input to "well-behaved"
    library functions that read from a "FILE *", and so on.

    In the OP's case, however, he is calling a "poorly-behaved" library
    function that uses a "global variable"[%], namely stdout. Since
    stdout need not be a modifiable lvalue, you cannot simply do:

    fflush(stdout);
    saved_stdout = stdout;
    stdout = fp;
    bad_library_function();
    fflush(stdout);
    stdout = saved_stdout;

    If you are willing to go for non-portable solutions, the sequence:

    target_fd = open_or_pipe_or_whatever();
    ... check for failures ...
    saved_stdout_fd = dup(STDOUT_FILENO);
    ... check for failures ...

    fflush(stdout);
    dup2(target_fd, STDOUT_FILENO);
    bad_library_function();
    fflush(stdout);
    dup2(saved_stdout_fd, STDOUT_FILENO);

    will do the trick on POSIX systems. The dup2() calls changes the
    "global variable"[%] STDOUT_FILENO "behind the back" of the C
    library stdio. The pre- and post-call fflush()es make sure that
    all stdout output goes to the "correct" fd. The non-portable
    version that reassigns the "global variable" stdout uses the same
    technique: push out all the "correct" output, make a "global
    variable" change, call the badly behaved function with the system's
    global state appropriately rearranged, push out *that* output, then
    rearrange the global state once more to put it back to normal.

    [% The words "global variable" are in quotes because C does not
    define the term, and in fact stdout and POSIX_FILENO may well be
    constants, not variables. In practice, stdout is either an "address
    constant", like &__sstdout, or an actual variable but possibly
    const-qualfied, while POSIX_FILENO is simply the integer constant
    1. Nonetheless, the POSIX stdout descriptor is a "global variable",
    by my definition. It is changeable via system call: dup2(fd, 1)
    changes it.]
    --
    In-Real-Life: Chris Torek, Wind River Systems
    Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
    email: gmail (figure it out) http://web.torek.net/torek/index.html
     
    Chris Torek, May 2, 2008
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Max
    Replies:
    7
    Views:
    9,124
  2. Peter A. Schott
    Replies:
    4
    Views:
    594
    Peter A. Schott
    Feb 7, 2006
  3. Replies:
    1
    Views:
    380
    Andrew Thompson
    Oct 23, 2006
  4. Replies:
    6
    Views:
    639
  5. Vivek
    Replies:
    3
    Views:
    622
    Thomas 'PointedEars' Lahn
    Apr 5, 2006
Loading...

Share This Page