Faster way to write in a file

L

LilacSkin

Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

Tk.
 
S

santosh

LilacSkin said:
Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}
 
L

LilacSkin

LilacSkin said:
Currently, I write an signed long long in a file with the fprintf
function:
signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;
pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);
for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}
The problem is that function is very very slow !
In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

It's not working !
 
S

santosh

LilacSkin said:
LilacSkin said:
Currently, I write an signed long long in a file with the fprintf
function:
signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;
pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);
for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}
The problem is that function is very very slow !
In several forums, I saw that fwrite function is better, but I
don't know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

It's not working !

How exactly? Did you check that the elements parameter is correct? What
is the value that rc has after the fwrite() call? Is pFilec pointing to
a valid stream for which you have appropriate permissions?
 
M

Mark Bluemel

LilacSkin said:
Hi,

Currently, I write an signed long long in a file with the fprintf
function:

No. You don't. You write the textual representation of the signed long
long value to the file. I will return to this point shortly.
signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

This snipped code is clearly incomplete, so tells us virtually nothing.

pDataRx is undefined. DataAvail is 0, so the loop will not be entered...
The problem is that function is very very slow !

Compared to what? How do you know?
In several forums, I saw that fwrite function is better,

Than what? In what way? Do you know (by measuring) that fprintf is the
cause of your problem?

As I pointed out above, fprintf is converting your signed long long
values to their textual representation. fwrite can't do that - it would
write the internal binary representation of the data.

I suppose you could write your own single-purpose function to convert
signed long long values into text and write that text with fwrite, and
that may perform better than fprintf...
but I don't know how to use it.

It (fwrite) is a simple enough function which is documented in the
standard and any good text.
Can you help me, please ?

You'd need to give us more information about the problem before we can
help you with the solution.
 
M

Mark Bluemel

santosh said:
LilacSkin said:
Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?

size_t rc;
rc = fwrite(pData, sizeof *pData, 1048576UL, pFilec);
if (rc != 1048576) {
puts("Write error.");
/* ... */
}

How does dumping the data in binary form compare to writing a
tab-delimited textual representation?
 
M

Morris Dovey

Mark said:
How does dumping the data in binary form compare to writing a
tab-delimited textual representation?

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -> text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp...6/995f74496633fe3a?lnk=st&q=#995f74496633fe3a

(mind the wrap)
 
M

Morris Dovey

Mark said:
My point was that the OP had clearly been trying to produce a textual
representation.

From context, I would not have expected the OP to understand why
Santosh's suggestion was not compatible with his/her original goal.

I agree with you - that's why I wrote what you snipped:

My thinking is that if speed is critical to LilacSkin, then the
textual representation can be produced somewhat faster (perhaps
almost twice as fast) by substituting a user-written function for
the standard formatting routine.

Once that's been done, fwrite() may indeed be an appropriate
mechanism. I'm actually not sure whether Santosh mentally skipped
a step or was advocating binary output. :)
 
M

Mark Bluemel

Morris said:
If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

My point was that the OP had clearly been trying to produce a textual
representation.

From context, I would not have expected the OP to understand why
Santosh's suggestion was not compatible with his/her original goal.
 
M

Malcolm McLean

LilacSkin said:
Hi,

Currently, I write an signed long long in a file with the fprintf
function:

signed long long * pData = NULL;
unsigned long long k = 0;
unsigned long DataAvail = 0 ;

pDataRx = (signed long long *) malloc ( sizeof(signed long long ) *
(65536*16*8) / 8);

for ( k = 0; k < DataAvail/8; k ++ ) {
fprintf ( pFilec, "%lli\t", pData[k]);
}

The problem is that function is very very slow !

In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?
Replace %lli with %llx and measure the speedup.
Is hexadecimal output acceptable to you?
 
S

santosh

Morris said:
I agree with you - that's why I wrote what you snipped:


My thinking is that if speed is critical to LilacSkin, then the
textual representation can be produced somewhat faster (perhaps
almost twice as fast) by substituting a user-written function for
the standard formatting routine.

Once that's been done, fwrite() may indeed be an appropriate
mechanism. I'm actually not sure whether Santosh mentally skipped
a step or was advocating binary output. :)

I was responding to the following text by the OP:

<quote>
In several forums, I saw that fwrite function is better, but I don't
know how to use it.
Can you help me, please ?
</quote>
 
L

LilacSkin

If the textual representation is being done by fprintf(), writing
the unconverted data in binary form is _much_ faster - and, of
course, may produce problems if there's a subsequent need to read
the data on a different machine.

A special-purpose double -> text conversion routine can provide a
fair amount of speed improvement. A real-world example was
developed in this CLC thread:

http://groups.google.com/group/comp.lang.c/browse_thread/thread/eb329...

(mind the wrap)

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

Thanks !
 
L

LilacSkin

Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

Thanks !

In fact the most important thing is to write faster.
I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.
The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

I think the best way is to format them and use a fwrite or _write
function .
 
W

Willem

LilacSkin wrote:
) In fact the most important thing is to write faster.
) I have 1,6 Mbyte/s of raw data to write during a couple of days.
) Because of the amount of data, I can't reformat their after.
) The data are stored in a char buffer and I need to put in a file as a
) signed long long.
) The size of the buzzer is 8 Mbyte, refreshed every 5 sec.

I assume 1,6 Mbyte/sec is the input speed.
Am I right that that's 200.000 numbers per second ?
I'm assuming one number takes about 16 bytes in the output file, so that's
actually 3,2 Mbyte/s to write to the file.

You can try having one thread formatting the block, and then have another
actually write the file. That way, you're not waiting for file I/O.

Here's some pseudocode for double-buffering, to give you an idea:
(Note that write() is not standard C.)

BUFSIZE = 4096;

buffer[BUFSIZE*2 + 32];
buf1 = buffer;
buf2 = buffer + BUFSIZE;
end = buffer + BUFSIZE*2;
flag = 0;

formatter_loop:
ptr = buf1;
while(1) {
while (ptr < buf2);
ptr += sprintf(ptr, "%lld\t", number);
while (flag)
/* wait */;
flag = buf1;
while (ptr < end)
ptr += sprintf(ptr, "%lld\t", number);
while (flag)
/* wait */;
flag = buf2;
memcpy(buf1, end, ptr - end);
ptr -= BUFSIZE*2;
}

writer_loop:
while(1) {
while (!flag)
/* wait */;
write(filedes, flag, BUFSIZE);
flag = 0;
}


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
M

Morris Dovey

LilacSkin said:
Your idea is to use your "convert" function and to fwrite the
converted buffer ?
If yes, can I cast an long long in double to directly use it ?

You probably won't want to use _that_ convert function because it
does more than you want (double -> string) when you're really
wanting (long long -> string).

Just throw away the part that deals with the fractional part of
the double - and if all of your values are positive, throw away
the sign logic, too. That won't leave very much - and it
shouldn't waste many cycles.
 
M

Morris Dovey

LilacSkin said:
#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = '0';
if (sign) *s = '-';
}

Pretty close. Since you're not working with a double, you can
dispense with the fractional part logic and shorten to something
like:

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}
 
L

LilacSkin

#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;

*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));

while (p >= s) *p-- = '0';

if (sign) *s = '-';

}

int main(void)
{ long long test = -123456789012345;
char buffer[64];

convert(test,buffer,25,5);
printf("'%s'\n",buffer);
// then fwrite(buffer...)

return 0;

}
 
B

Ben Bacarisse

LilacSkin said:
#include <stdio.h>

void convert(long long v,char *s,int sz,int dp)
{ char *p = s + sz;
long long x = 1LL;
int sign = v < 0.0;

if (sign) v = -v;
while (dp--) x *= 10;
x = x * v;

*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));

while (p >= s) *p-- = '0';

if (sign) *s = '-';

}

int main(void)
{ long long test = -123456789012345;
char buffer[64];

convert(test,buffer,25,5);
printf("'%s'\n",buffer);
// then fwrite(buffer...)

return 0;

}

You may want to re-consider. Numbers are usually converted to a
decimal string representation so the we (humans) can read them. If
your program is producing such a volume of data that it is hard to get
converted output fast enough, will the resulting data ever be read by
a person? I doubt it. You'd need a program to scan just one second's
worth.

It might pay to store the data as native binary numbers. That is
probably what was being suggested when someone said use "fwrite".

Of course, if the output *has* to be processed by an existing program
that needs decimal input, then the conversion has to happen some time,
but it could be done later, at leisure, so to speak. It might even be
possible to process into decimal only the portion you are interested in.
Maybe the binary data could be indexed for rapid access to specific
parts. Without knowing the ultimate fate of this data, it is hard to
be more specific, but I suggest you look beyond your current headache
of not being able to use fprintf because it seems too slow.
 
M

Mark Bluemel

LilacSkin said:
I have 1,6 Mbyte/s of raw data to write during a couple of days.
Because of the amount of data, I can't reformat their after.
The data are stored in a char buffer and I need to put in a file as a
signed long long.

So why were you even trying to use printf, which is for formatted
output?

If you have to write to the file as signed long long, then fwrite is
appropriate.

You really could help us to help you if you explained the requirement
better.
 
L

LilacSkin

So why were you even trying to use printf, which is for formatted
output?

If you have to write to the file as signed long long, then fwrite is
appropriate.

You really could help us to help you if you explained the requirement
better.

I need to write the data in text file !
Morris, I try your code, the problem is, it puts a space character
before a positive number.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top