Faster way to write in a file

Discussion in 'C Programming' started by LilacSkin, Feb 13, 2008.

  1. LilacSkin

    LilacSkin Guest

    Hi,

    Currently, I write an signed long long in a file with the fprintf
    function:

    signed long long * pData = NULL;
    unsigned long long k = 0;
    unsigned long DataAvail = 0 ;

    pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    (65536*16*8) / 8);

    for ( k = 0; k < DataAvail/8; k ++ ) {
    fprintf ( pFilec, "%lli\t", pData[k]);
    }

    The problem is that function is very very slow !

    In several forums, I saw that fwrite function is better, but I don't
    know how to use it.
    Can you help me, please ?

    Tk.
     
    LilacSkin, Feb 13, 2008
    #1
    1. Advertising

  2. LilacSkin

    santosh Guest

    LilacSkin wrote:

    > Hi,
    >
    > Currently, I write an signed long long in a file with the fprintf
    > function:
    >
    > signed long long * pData = NULL;
    > unsigned long long k = 0;
    > unsigned long DataAvail = 0 ;
    >
    > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    > (65536*16*8) / 8);
    >
    > for ( k = 0; k < DataAvail/8; k ++ ) {
    > fprintf ( pFilec, "%lli\t", pData[k]);
    > }
    >
    > The problem is that function is very very slow !
    >
    > In several forums, I saw that fwrite function is better, but I don't
    > know how to use it.
    > Can you help me, please ?


    size_t rc;
    rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
    if (rc != 1048576) {
    puts("Write error.");
    /* ... */
    }
     
    santosh, Feb 13, 2008
    #2
    1. Advertising

  3. LilacSkin

    LilacSkin Guest

    On 13 fév, 13:47, santosh <> wrote:
    > LilacSkin wrote:
    > > Hi,

    >
    > > Currently, I write an signed long long in a file with the fprintf
    > > function:

    >
    > > signed long long * pData = NULL;
    > > unsigned long long k = 0;
    > > unsigned long DataAvail = 0 ;

    >
    > > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    > > (65536*16*8) / 8);

    >
    > > for ( k = 0; k < DataAvail/8; k ++ ) {
    > > fprintf ( pFilec, "%lli\t", pData[k]);
    > > }

    >
    > > The problem is that function is very very slow !

    >
    > > In several forums, I saw that fwrite function is better, but I don't
    > > know how to use it.
    > > Can you help me, please ?

    >
    > size_t rc;
    > rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
    > if (rc != 1048576) {
    > puts("Write error.");
    > /* ... */
    > }


    It's not working !
     
    LilacSkin, Feb 13, 2008
    #3
  4. LilacSkin

    santosh Guest

    LilacSkin wrote:

    > On 13 fév, 13:47, santosh <> wrote:
    >> LilacSkin wrote:
    >> > Hi,

    >>
    >> > Currently, I write an signed long long in a file with the fprintf
    >> > function:

    >>
    >> > signed long long * pData = NULL;
    >> > unsigned long long k = 0;
    >> > unsigned long DataAvail = 0 ;

    >>
    >> > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    >> > (65536*16*8) / 8);

    >>
    >> > for ( k = 0; k < DataAvail/8; k ++ ) {
    >> > fprintf ( pFilec, "%lli\t", pData[k]);
    >> > }

    >>
    >> > The problem is that function is very very slow !

    >>
    >> > In several forums, I saw that fwrite function is better, but I
    >> > don't know how to use it.
    >> > Can you help me, please ?

    >>
    >> size_t rc;
    >> rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
    >> if (rc != 1048576) {
    >> puts("Write error.");
    >> /* ... */
    >> }

    >
    > It's not working !


    How exactly? Did you check that the elements parameter is correct? What
    is the value that rc has after the fwrite() call? Is pFilec pointing to
    a valid stream for which you have appropriate permissions?
     
    santosh, Feb 13, 2008
    #4
  5. LilacSkin

    Mark Bluemel Guest

    LilacSkin wrote:
    > Hi,
    >
    > Currently, I write an signed long long in a file with the fprintf
    > function:


    No. You don't. You write the textual representation of the signed long
    long value to the file. I will return to this point shortly.

    > signed long long * pData = NULL;
    > unsigned long long k = 0;
    > unsigned long DataAvail = 0 ;
    >
    > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    > (65536*16*8) / 8);
    >
    > for ( k = 0; k < DataAvail/8; k ++ ) {
    > fprintf ( pFilec, "%lli\t", pData[k]);
    > }


    This snipped code is clearly incomplete, so tells us virtually nothing.

    pDataRx is undefined. DataAvail is 0, so the loop will not be entered...

    > The problem is that function is very very slow !


    Compared to what? How do you know?

    > In several forums, I saw that fwrite function is better,


    Than what? In what way? Do you know (by measuring) that fprintf is the
    cause of your problem?

    As I pointed out above, fprintf is converting your signed long long
    values to their textual representation. fwrite can't do that - it would
    write the internal binary representation of the data.

    I suppose you could write your own single-purpose function to convert
    signed long long values into text and write that text with fwrite, and
    that may perform better than fprintf...

    > but I don't know how to use it.


    It (fwrite) is a simple enough function which is documented in the
    standard and any good text.

    > Can you help me, please ?


    You'd need to give us more information about the problem before we can
    help you with the solution.
     
    Mark Bluemel, Feb 13, 2008
    #5
  6. LilacSkin

    Mark Bluemel Guest

    santosh wrote:
    > LilacSkin wrote:
    >
    >> Hi,
    >>
    >> Currently, I write an signed long long in a file with the fprintf
    >> function:
    >>
    >> signed long long * pData = NULL;
    >> unsigned long long k = 0;
    >> unsigned long DataAvail = 0 ;
    >>
    >> pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    >> (65536*16*8) / 8);
    >>
    >> for ( k = 0; k < DataAvail/8; k ++ ) {
    >> fprintf ( pFilec, "%lli\t", pData[k]);
    >> }
    >>
    >> The problem is that function is very very slow !
    >>
    >> In several forums, I saw that fwrite function is better, but I don't
    >> know how to use it.
    >> Can you help me, please ?

    >
    > size_t rc;
    > rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
    > if (rc != 1048576) {
    > puts("Write error.");
    > /* ... */
    > }


    How does dumping the data in binary form compare to writing a
    tab-delimited textual representation?
     
    Mark Bluemel, Feb 13, 2008
    #6
  7. LilacSkin

    Morris Dovey Guest

    Mark Bluemel wrote:

    > How does dumping the data in binary form compare to writing a
    > tab-delimited textual representation?


    If the textual representation is being done by fprintf(), writing
    the unconverted data in binary form is _much_ faster - and, of
    course, may produce problems if there's a subsequent need to read
    the data on a different machine.

    A special-purpose double -> text conversion routine can provide a
    fair amount of speed improvement. A real-world example was
    developed in this CLC thread:

    http://groups.google.com/group/comp...6/995f74496633fe3a?lnk=st&q=#995f74496633fe3a

    (mind the wrap)

    --
    Morris Dovey
    DeSoto Solar
    DeSoto, Iowa USA
    http://www.iedu.com/DeSoto
     
    Morris Dovey, Feb 13, 2008
    #7
  8. LilacSkin

    Morris Dovey Guest

    Mark Bluemel wrote:

    > My point was that the OP had clearly been trying to produce a textual
    > representation.
    >
    > From context, I would not have expected the OP to understand why
    > Santosh's suggestion was not compatible with his/her original goal.


    I agree with you - that's why I wrote what you snipped:

    >> A special-purpose double -> text conversion routine can provide a
    >> fair amount of speed improvement.


    My thinking is that if speed is critical to LilacSkin, then the
    textual representation can be produced somewhat faster (perhaps
    almost twice as fast) by substituting a user-written function for
    the standard formatting routine.

    Once that's been done, fwrite() may indeed be an appropriate
    mechanism. I'm actually not sure whether Santosh mentally skipped
    a step or was advocating binary output. :)

    --
    Morris Dovey
    DeSoto Solar
    DeSoto, Iowa USA
    http://www.iedu.com/DeSoto
     
    Morris Dovey, Feb 13, 2008
    #8
  9. LilacSkin

    Mark Bluemel Guest

    Morris Dovey wrote:
    > Mark Bluemel wrote:
    >
    >> How does dumping the data in binary form compare to writing a
    >> tab-delimited textual representation?

    >
    > If the textual representation is being done by fprintf(), writing
    > the unconverted data in binary form is _much_ faster - and, of
    > course, may produce problems if there's a subsequent need to read
    > the data on a different machine.


    My point was that the OP had clearly been trying to produce a textual
    representation.

    From context, I would not have expected the OP to understand why
    Santosh's suggestion was not compatible with his/her original goal.
     
    Mark Bluemel, Feb 13, 2008
    #9
  10. "LilacSkin" <> wrote in message news
    > Hi,
    >
    > Currently, I write an signed long long in a file with the fprintf
    > function:
    >
    > signed long long * pData = NULL;
    > unsigned long long k = 0;
    > unsigned long DataAvail = 0 ;
    >
    > pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
    > (65536*16*8) / 8);
    >
    > for ( k = 0; k < DataAvail/8; k ++ ) {
    > fprintf ( pFilec, "%lli\t", pData[k]);
    > }
    >
    > The problem is that function is very very slow !
    >
    > In several forums, I saw that fwrite function is better, but I don't
    > know how to use it.
    > Can you help me, please ?
    >

    Replace %lli with %llx and measure the speedup.
    Is hexadecimal output acceptable to you?

    --
    Free games and programming goodies.
    http://www.personal.leeds.ac.uk/~bgy1mm
     
    Malcolm McLean, Feb 13, 2008
    #10
  11. LilacSkin

    santosh Guest

    Morris Dovey wrote:

    > Mark Bluemel wrote:
    >
    >> My point was that the OP had clearly been trying to produce a textual
    >> representation.
    >>
    >> From context, I would not have expected the OP to understand why
    >> Santosh's suggestion was not compatible with his/her original goal.

    >
    > I agree with you - that's why I wrote what you snipped:
    >
    >>> A special-purpose double -> text conversion routine can provide a
    >>> fair amount of speed improvement.

    >
    > My thinking is that if speed is critical to LilacSkin, then the
    > textual representation can be produced somewhat faster (perhaps
    > almost twice as fast) by substituting a user-written function for
    > the standard formatting routine.
    >
    > Once that's been done, fwrite() may indeed be an appropriate
    > mechanism. I'm actually not sure whether Santosh mentally skipped
    > a step or was advocating binary output. :)


    I was responding to the following text by the OP:

    <quote>
    In several forums, I saw that fwrite function is better, but I don't
    know how to use it.
    Can you help me, please ?
    </quote>
     
    santosh, Feb 13, 2008
    #11
  12. LilacSkin

    LilacSkin Guest

    On 13 fév, 15:13, Morris Dovey <> wrote:
    > Mark Bluemel wrote:
    > > How does dumping the data in binary form compare to writing a
    > > tab-delimited textual representation?

    >
    > If the textual representation is being done by fprintf(), writing
    > the unconverted data in binary form is _much_ faster - and, of
    > course, may produce problems if there's a subsequent need to read
    > the data on a different machine.
    >
    > A special-purpose double -> text conversion routine can provide a
    > fair amount of speed improvement. A real-world example was
    > developed in this CLC thread:
    >
    > http://groups.google.com/group/comp.lang.c/browse_thread/thread/eb329...
    >
    > (mind the wrap)
    >
    > --
    > Morris Dovey
    > DeSoto Solar
    > DeSoto, Iowa USAhttp://www.iedu.com/DeSoto


    Your idea is to use your "convert" function and to fwrite the
    converted buffer ?
    If yes, can I cast an long long in double to directly use it ?

    Thanks !
     
    LilacSkin, Feb 13, 2008
    #12
  13. LilacSkin

    LilacSkin Guest

    On 13 fév, 19:51, LilacSkin <> wrote:
    > On 13 fév, 15:13, Morris Dovey <> wrote:
    >
    >
    >
    > > Mark Bluemel wrote:
    > > > How does dumping the data in binary form compare to writing a
    > > > tab-delimited textual representation?

    >
    > > If the textual representation is being done by fprintf(), writing
    > > the unconverted data in binary form is _much_ faster - and, of
    > > course, may produce problems if there's a subsequent need to read
    > > the data on a different machine.

    >
    > > A special-purpose double -> text conversion routine can provide a
    > > fair amount of speed improvement. A real-world example was
    > > developed in this CLC thread:

    >
    > >http://groups.google.com/group/comp.lang.c/browse_thread/thread/eb329...

    >
    > > (mind the wrap)

    >
    > > --
    > > Morris Dovey
    > > DeSoto Solar
    > > DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

    >
    > Your idea is to use your "convert" function and to fwrite the
    > converted buffer ?
    > If yes, can I cast an long long in double to directly use it ?
    >
    > Thanks !


    In fact the most important thing is to write faster.
    I have 1,6 Mbyte/s of raw data to write during a couple of days.
    Because of the amount of data, I can't reformat their after.
    The data are stored in a char buffer and I need to put in a file as a
    signed long long.
    The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

    I think the best way is to format them and use a fwrite or _write
    function .
     
    LilacSkin, Feb 13, 2008
    #13
  14. LilacSkin

    Willem Guest

    LilacSkin wrote:
    ) In fact the most important thing is to write faster.
    ) I have 1,6 Mbyte/s of raw data to write during a couple of days.
    ) Because of the amount of data, I can't reformat their after.
    ) The data are stored in a char buffer and I need to put in a file as a
    ) signed long long.
    ) The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

    I assume 1,6 Mbyte/sec is the input speed.
    Am I right that that's 200.000 numbers per second ?
    I'm assuming one number takes about 16 bytes in the output file, so that's
    actually 3,2 Mbyte/s to write to the file.

    You can try having one thread formatting the block, and then have another
    actually write the file. That way, you're not waiting for file I/O.

    Here's some pseudocode for double-buffering, to give you an idea:
    (Note that write() is not standard C.)

    BUFSIZE = 4096;

    buffer[BUFSIZE*2 + 32];
    buf1 = buffer;
    buf2 = buffer + BUFSIZE;
    end = buffer + BUFSIZE*2;
    flag = 0;

    formatter_loop:
    ptr = buf1;
    while(1) {
    while (ptr < buf2);
    ptr += sprintf(ptr, "%lld\t", number);
    while (flag)
    /* wait */;
    flag = buf1;
    while (ptr < end)
    ptr += sprintf(ptr, "%lld\t", number);
    while (flag)
    /* wait */;
    flag = buf2;
    memcpy(buf1, end, ptr - end);
    ptr -= BUFSIZE*2;
    }

    writer_loop:
    while(1) {
    while (!flag)
    /* wait */;
    write(filedes, flag, BUFSIZE);
    flag = 0;
    }


    SaSW, Willem
    --
    Disclaimer: I am in no way responsible for any of the statements
    made in the above text. For all I know I might be
    drugged or something..
    No I'm not paranoid. You all think I'm paranoid, don't you !
    #EOT
     
    Willem, Feb 13, 2008
    #14
  15. LilacSkin

    Morris Dovey Guest

    LilacSkin wrote:
    >
    > On 13 fév, 15:13, Morris Dovey <> wrote:
    > > Mark Bluemel wrote:
    > > > How does dumping the data in binary form compare to writing a
    > > > tab-delimited textual representation?

    > >
    > > If the textual representation is being done by fprintf(), writing
    > > the unconverted data in binary form is _much_ faster - and, of
    > > course, may produce problems if there's a subsequent need to read
    > > the data on a different machine.
    > >
    > > A special-purpose double -> text conversion routine can provide a
    > > fair amount of speed improvement. A real-world example was
    > > developed in this CLC thread:
    > >
    > > http://groups.google.com/group/comp.lang.c/browse_thread/thread/eb329...
    > >
    > > (mind the wrap)
    > >
    > > --
    > > Morris Dovey
    > > DeSoto Solar
    > > DeSoto, Iowa USAhttp://www.iedu.com/DeSoto

    >
    > Your idea is to use your "convert" function and to fwrite the
    > converted buffer ?
    > If yes, can I cast an long long in double to directly use it ?


    You probably won't want to use _that_ convert function because it
    does more than you want (double -> string) when you're really
    wanting (long long -> string).

    Just throw away the part that deals with the fractional part of
    the double - and if all of your values are positive, throw away
    the sign logic, too. That won't leave very much - and it
    shouldn't waste many cycles.

    --
    Morris Dovey
    DeSoto Solar
    DeSoto, Iowa USA
    http://www.iedu.com/DeSoto
     
    Morris Dovey, Feb 13, 2008
    #15
  16. LilacSkin

    Morris Dovey Guest

    LilacSkin wrote:

    > #include <stdio.h>
    >
    > void convert(long long v,char *s,int sz,int dp)
    > { char *p = s + sz;
    > long long x = 1LL;
    > int sign = v < 0.0;
    >
    > if (sign) v = -v;
    > while (dp--) x *= 10;
    > x = x * v;
    > *p-- = '\0';
    > do
    > { *p-- = '0' + (x % 10);
    > } while ((p >= s) && (x /= 10));
    > while (p >= s) *p-- = '0';
    > if (sign) *s = '-';
    > }


    Pretty close. Since you're not working with a double, you can
    dispense with the fractional part logic and shorten to something
    like:

    void convert(long long x,char *s,int sz)
    { char *p = s + sz;
    int sign = x < 0;

    if (sign) x = -x;
    *p-- = '\0';
    do
    { *p-- = '0' + (x % 10);
    } while ((p >= s) && (x /= 10));
    while (p >= s) *p-- = ' ';
    if (sign) *s = '-';
    }

    --
    Morris Dovey
    DeSoto Solar
    DeSoto, Iowa USA
    http://www.iedu.com/DeSoto
     
    Morris Dovey, Feb 13, 2008
    #16
  17. LilacSkin

    LilacSkin Guest

    #include <stdio.h>

    void convert(long long v,char *s,int sz,int dp)
    { char *p = s + sz;
    long long x = 1LL;
    int sign = v < 0.0;

    if (sign) v = -v;
    while (dp--) x *= 10;
    x = x * v;

    *p-- = '\0';
    do
    { *p-- = '0' + (x % 10);
    } while ((p >= s) && (x /= 10));

    while (p >= s) *p-- = '0';

    if (sign) *s = '-';

    }

    int main(void)
    { long long test = -123456789012345;
    char buffer[64];

    convert(test,buffer,25,5);
    printf("'%s'\n",buffer);
    // then fwrite(buffer...)

    return 0;

    }
     
    LilacSkin, Feb 13, 2008
    #17
  18. LilacSkin <> writes:

    > #include <stdio.h>
    >
    > void convert(long long v,char *s,int sz,int dp)
    > { char *p = s + sz;
    > long long x = 1LL;
    > int sign = v < 0.0;
    >
    > if (sign) v = -v;
    > while (dp--) x *= 10;
    > x = x * v;
    >
    > *p-- = '\0';
    > do
    > { *p-- = '0' + (x % 10);
    > } while ((p >= s) && (x /= 10));
    >
    > while (p >= s) *p-- = '0';
    >
    > if (sign) *s = '-';
    >
    > }
    >
    > int main(void)
    > { long long test = -123456789012345;
    > char buffer[64];
    >
    > convert(test,buffer,25,5);
    > printf("'%s'\n",buffer);
    > // then fwrite(buffer...)
    >
    > return 0;
    >
    > }


    You may want to re-consider. Numbers are usually converted to a
    decimal string representation so the we (humans) can read them. If
    your program is producing such a volume of data that it is hard to get
    converted output fast enough, will the resulting data ever be read by
    a person? I doubt it. You'd need a program to scan just one second's
    worth.

    It might pay to store the data as native binary numbers. That is
    probably what was being suggested when someone said use "fwrite".

    Of course, if the output *has* to be processed by an existing program
    that needs decimal input, then the conversion has to happen some time,
    but it could be done later, at leisure, so to speak. It might even be
    possible to process into decimal only the portion you are interested in.
    Maybe the binary data could be indexed for rapid access to specific
    parts. Without knowing the ultimate fate of this data, it is hard to
    be more specific, but I suggest you look beyond your current headache
    of not being able to use fprintf because it seems too slow.

    --
    Ben.
     
    Ben Bacarisse, Feb 13, 2008
    #18
  19. LilacSkin

    Mark Bluemel Guest

    LilacSkin wrote:

    > I have 1,6 Mbyte/s of raw data to write during a couple of days.
    > Because of the amount of data, I can't reformat their after.
    > The data are stored in a char buffer and I need to put in a file as a
    > signed long long.


    So why were you even trying to use printf, which is for formatted
    output?

    If you have to write to the file as signed long long, then fwrite is
    appropriate.

    You really could help us to help you if you explained the requirement
    better.
     
    Mark Bluemel, Feb 14, 2008
    #19
  20. LilacSkin

    LilacSkin Guest

    On 14 fév, 09:59, Mark Bluemel <> wrote:
    > LilacSkin wrote:
    > > I have 1,6 Mbyte/s of raw data to write during a couple of days.
    > > Because of the amount of data, I can't reformat their after.
    > > The data are stored in a char buffer and I need to put in a file as a
    > > signed long long.

    >
    > So why were you even trying to use printf, which is for formatted
    > output?
    >
    > If you have to write to the file as signed long long, then fwrite is
    > appropriate.
    >
    > You really could help us to help you if you explained the requirement
    > better.


    I need to write the data in text file !
    Morris, I try your code, the problem is, it puts a space character
    before a positive number.
     
    LilacSkin, Feb 14, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Terry Olsen

    Any way to make pages faster?

    Terry Olsen, Aug 1, 2005, in forum: ASP .Net
    Replies:
    6
    Views:
    379
    clintonG
    Aug 2, 2005
  2. Matt Stephens

    XSLT is way faster using Java 5

    Matt Stephens, Feb 6, 2005, in forum: Java
    Replies:
    0
    Views:
    410
    Matt Stephens
    Feb 6, 2005
  3. steve
    Replies:
    17
    Views:
    700
    Mike Smith
    Sep 13, 2004
  4. Jon Hyland
    Replies:
    4
    Views:
    6,089
    Jon Hyland
    Oct 5, 2004
  5. Eamonn Sullivan

    A faster way of finding historical highs/lows

    Eamonn Sullivan, Jun 11, 2004, in forum: Python
    Replies:
    6
    Views:
    360
    Peter Hansen
    Jun 14, 2004
Loading...

Share This Page