Faster way to write in a file

M

Mark Bluemel

LilacSkin said:
I need to write the data in text file !

OK. So what is the required format of this text file?

We can assume decimal numbers, but do you want fixed or variable size?
If fixed size, do you want blank-padded or leading zeros?
Do you want new-lines at some interval? If so, what interval?
Morris, I try your code, the problem is, it puts a space character
before a positive number.

Why shouldn't it? You didn't indicate that you wanted otherwise.
 
L

LilacSkin

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}


time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.


NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1
 
L

LilacSkin

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}


time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.


NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1
 
L

LilacSkin

The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}


time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.


NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

1
 
M

Mark Bluemel

LilacSkin said:
The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );

time ( &start_time );

for (i=0;i<16000000;i++)
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
if (i % 16 == 15) fputc ( (int) '\n' , pFile );
else fputc ( (int) '\t' , pFile );
}


time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.

Easy fix :-

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
else *s = '+';
}

if you don't want a sign character at all, then a crude approach could
be :-
if (test < 0) fwrite (buffer , 1 , num_digit, pFile );
else fwrite (buffer + 1, 1, num_digit-1, pFile);

More elegant solutions are obviously possible, but for quick simple
hacks, these may work.
 
B

Ben Bacarisse

LilacSkin said:
The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
}
The problem is, there is a space character before a positive number.

NEEDED: 1 line each 80 us
REACHED: 1 line each 36 us

Form a quick test here, it seems faster (almost twice the speed) to
reverse the string after getting the digits:

int convert2(long long x, char *s)
{
char *p = s;
int nd, sign = x < 0;
if (sign) x = -x;
do {
*p++ = '0' + (x % 10);
x /= 10;
} while (x);
if (sign) *p++ = '-';
*p = 0;
nd = p - s;
while (--p > s) {
char c = *s;
*s++ = *p;
*p = c;
}
return nd;
}

You get the digit count almost for free that way.
 
A

Army1987

You posted this three times in two minutes. Before assuming that a posting
failed and retrying to post the same article you should allow some time
for the post to arrive.
LilacSkin said:
The appropriate format is :

(1st signed long long) (tab) (2nd signed long long) (tab) ... (16th
signed long long) (\n)
(17th signed long long) .....

what I've done :


#include <stdio.h>
#include <time.h>

void convert(long long x,char *s,int sz)
{ char *p = s + sz;
{ char *p = s + sz;
int sign = x < 0;

if (sign) x = -x;
*p-- = '\0';
do
{ *p-- = '0' + (x % 10);
} while ((p >= s) && (x /= 10));
while (p >= s) *p-- = ' ';
if (sign) *s = '-';
}

int digits(long long x)
{ int n = 1;
do
{ n++;
} while (x /= 10);
return n;
} See comments at the bottom.

int main(void)
{
time_t start_time;
time_t end_time;
double diff_time = 0;

int i;
int num_digit;
char buffer[64];
long long test = -1844674407370955161;
FILE * pFile = NULL;
pFile = fopen( "data_test_write.txt", "w" );
How do you know that it didn't fail?
time ( &start_time );

for (i=0;i<16000000;i++)
That isn't required to fit in an int. If you mind portability you should
declare i as long if you know it's going to hold numbers > 32767.
{
num_digit=digits(test);
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.
if (i % 16 == 15) fputc ((int) '\n' , pFile );
That cast is useless. '\n' already has type int, and if it hadn't it would
be converted to one.
else fputc ((int) '\t' , pFile );
Look up the ?: operator. While abusing it can make code unreadable,
moderately using it can be handy when otherwise you'd have almost
identical if and else branches.
fputc( (i % 16 == 15) ? '\n' : '\t', pFile);
}


time ( &end_time );
diff_time = difftime ( end_time, start_time );
printf("Duration : %lf sec\n", diff_time);
printf("Num of samples (16*64bit + /t and /n) : %i\n", i/16);

return 0;
}

The problem is, there is a space character before a positive number.

Your digits() function return a number too large by one. For negative
numbers, the extra character is used by the minus sign.
 
K

Keith Thompson

Army1987 said:
LilacSkin wrote: [...]
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.
[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.
 
J

Joe Wright

Keith said:
Army1987 said:
LilacSkin wrote: [...]
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.
[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.

I'd think that file I/O would take so much time that the performance of
conversion routines would be imperceptible.
 
K

Keith Thompson

Joe Wright said:
Keith said:
Army1987 said:
LilacSkin wrote: [...]
convert(test,buffer,num_digit);
//printf("'%s'\n",buffer);
fwrite (buffer , 1 , num_digit, pFile );
You can use fprintf with "lld":
fprintf(pFile, "%lld", test);
Its implementer probably has used the fastest possible routine to convert
a number to a decimal representation on your system.
[...]

True, but printf also has to parse the format string. A hand-written
conversion routine could easily beat printf's performance because of
this.

I'd think that file I/O would take so much time that the performance
of conversion routines would be imperceptible.

Possibly. The answer, of course, is to perform manual optimizations
if and only if the performance is measurably poor and the optimization
significantly improves it.

Note also that output is usually buffered, so a given fprintf call
might not result in any physical I/O at all. Or you might be using
sprintf rather than fprintf, or the target file might be on a RAM
disk, or ....
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,481
Members
44,900
Latest member
Nell636132

Latest Threads

Top