A question about reading an UTF-8 text file

C

Claude Yih

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :


int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer));

if ((fp = fopen(FILE_NAME, "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
}
}
else
{
}
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("ã‚ã„ã†ãˆãŠ") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well，The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~
 
B

Barry Schwarz

Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :


int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer));

if ((fp = fopen(FILE_NAME, "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
Click to expand...

The cast is superfluous.
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
Click to expand...

%x requires an unsigned int. On your system, char is apparently
signed. Try changing cp to an unsigned char* and buffer to an
unsigned char array.
}
}
else
{
}
Click to expand...

If you don't have any else action, you don't need the else at all.
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("?????") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well?The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~
Click to expand...

Since printf is a variadic function, all integer arguments smaller
than int (unsigned int) are promoted to int (or unsigned int). Think
about why 0x81 has the Fs but 0x61 doesn't.

Similarly, all floats would be promoted to double.


Remove del for email
 
K

Keith Thompson

Claude Yih said:
Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :


int main(void)
{
Click to expand...
[snip]

Why would you want to ignore the #include and #define directives?
Your program is invalid without them. If you're going to post a
program, post the whole thing (unless it's too big to post).
 
M

Martin Ambuhl

Claude said:
Hi, everyone. The other day I wrote a piece of code to see how the
UTF-8 text file was saved in the memory. Although the code worked
seemingly well, I still cannot understand a problem about the output
format.

The code is like this (I ignored the #include and #define here) :
Click to expand...

I won't ignore #includes and #defines. It is expected that code posted
here be compilable. The remainder of Claude Yih's question follows this
note.
See what the following does on your implementation. Note the 'unsigned'.

#include <stdio.h>
#define FILE_NAME "./japchar"

int main(void)
{
FILE *fp = NULL;
unsigned char buffer[BUFSIZ], *cp;

if ((fp = fopen(FILE_NAME, "rb"))) {
if (fread(buffer, 1, BUFSIZ, fp))
for (cp = buffer; *cp; cp++)
printf("%#X\t", *cp);
putchar('\n');
fclose(fp);
}
else
perror("open");

return 0;
}

[output for my implementation with ./japchar containing the characters
ã‚ã„ã†ãˆãŠ + EOL characters]

0XE3 0X81 0X82 0XE3 0X81 0X84 0XE3 0X81 0X86
0XE3 0X81 0X88 0XE3 0X81 0X8A 0XD 0XA

int main(void)
{
FILE* fp = NULL;
char buffer[MAX_SIZE];
int byteRead = 0;
char* cp = NULL;

memset(buffer, '\0', sizeof(buffer));

if ((fp = fopen(FILE_NAME, "r")) != NULL)
{
byteRead = fread(buffer, sizeof(char), MAX_SIZE, fp);
if (byteRead != 0)
{
cp = (char *)buffer;
while ((*cp) != '\0')
{
printf("%#X\t", *cp++);
}
}
else
{
}
fclose(fp);
}
else
{
perror("open");
}
return 0;
}

The UTF-8 text file it reads in contains five Japanese characters
("ã‚ã„ã†ãˆãŠ") and four ascii character ("abcd"). And the output
is as below:

0XFFFFFFEF 0XFFFFFFBB 0XFFFFFFBF 0XFFFFFFE3
0XFFFFFF81
0XFFFFFF82 0XFFFFFFE3 0XFFFFFF81 0XFFFFFF84
0XFFFFFFE3
0XFFFFFF81 0XFFFFFF86 0XFFFFFFE3 0XFFFFFF81
0XFFFFFF88
0XFFFFFFE3 0XFFFFFF81 0XFFFFFF8A 0X61 0X62 0X63
0X64

Well，The result is fine, but I don't understand how do those "FFFFFF"
come out? In my opinion, printf("%#X\t", *cp++) will only output the
content of one byte, but the output seems like four bytes. I don't know
why. Can anybody tell my how does it happen? Thanx~~~
Click to expand...
 
C

Claude Yih

Yeh, referring to the <Computer Systems - A Programmer's Perspective>,
I knew why I should change the char into unsigned char here. Thanx,
buddies :)

P.S: Next time, I won't ignore #include and #define any more :) Thanks
for reminding me of that.
 
V

Vladimir S. Oka

Claude Yih opined:
Yeh, referring to the <Computer Systems - A Programmer's
Perspective>, I knew why I should change the char into
unsigned char here. Thanx, buddies :)
Click to expand...

But we don't, as you didn't post any context. Read:

<http://cfaj.freeshell.org/google/>
P.S: Next time, I won't ignore #include and #define any more
:) Thanks for reminding me of that.
Click to expand...

Also, don't forget to follow good advice at the above URLs.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Reading text file contents to a character buffer 29
Cyrillic text from file - set utf8 in cmd, unknown characters output anyway 0
MeCab UTF-8 Decoding Problem 6
ANSI/UTF-8 File when save string to it 4
printf and UTF-8 in linux 6
[ruby 1.9] reading an UTF-8 encoded file 12
Read utf-8 file return utf-16 coding hex string ? 18
[Python3] Reading a binary file and wrtiting the bytes verbatim in an utf-8 file 6

Members online

No members online now.
Total: 29 (members: 0, guests: 29)
Robots: 229

Forum statistics

Threads
473,755
Messages
2,569,537
Members
45,022
Latest member
MaybelleMa

Latest Threads

Top