scanning UTF-8 characters

[Kamal R. Prasad]

Hello,

I am using a lexer (lex specification supplied to lex) to parse data,
and one of the requirements is to handle UTF-8 characters. My
understanding is that the first non-ascii character's byte will be >
0x7f in a UTF-8 character If I look for the same in yytext -will that
suffice? Is there some std function that one can use to operate on the
input stream? I want my code to be locale agnostic.

thanks
-kamal

 

[Jack Klein]

Hello,

I am using a lexer (lex specification supplied to lex) to parse data,
and one of the requirements is to handle UTF-8 characters. My
understanding is that the first non-ascii character's byte will be >
0x7f in a UTF-8 character If I look for the same in yytext -will that
suffice? Is there some std function that one can use to operate on the
input stream? I want my code to be locale agnostic.

thanks
-kamal

Neither lex nor UTF-8 is defined by the C language. Information on
UTF-8 can be obtained from http://www.unicode.org. Questions about
lex can be asked in

 

[Micah Cowan]

Hello,

I am using a lexer (lex specification supplied to lex) to parse data,
and one of the requirements is to handle UTF-8 characters. My
understanding is that the first non-ascii character's byte will be >
0x7f in a UTF-8 character If I look for the same in yytext -will that
suffice? Is there some std function that one can use to operate on the
input stream? I want my code to be locale agnostic.

Not really topical here in clc and clcm, I'm afraid. I've redirected
to comp.unix.programmer, where I believe you'll find more people able
to answer your question.

The /first/ non-ascii character's byte will be > 0xC0. But, yeah, you
should test for the high-bit. /All/ of the bytes in a
non-single-byte-character will be greater than 0x7f. The first byte
also has encoded information about how many bytes there are, total,
for this character.

As to how this fits in with lex, I'm not really qualified to say
much. Is it sufficient to look for the high bit? It depends on what
you intend to do after you've found one. And to be locale agnostic,
you'll probably need something to convert the locale's encoding into
UTF8 before scanning.

 

[Yang Jiao]

I don't know if a lexer (for me, I get flex in hand) could do anything to
identify the UTF-8 char, I m afraid u should do the job by ur own code.

 

[Jasen Betts]

["Followup-To:" header set to comp.lang.c.moderated.]

Hello,

I am using a lexer (lex specification supplied to lex) to parse data,
and one of the requirements is to handle UTF-8 characters. My
understanding is that the first non-ascii character's byte will be >
0x7f in a UTF-8 character If I look for the same in yytext -will that
suffice?

in most cases, there's a thing called windowing that can IIRC substitute
other symbols into the 0x00 to 0x7f range.

Is there some std function that one can use to operate on the
input stream? I want my code to be locale agnostic.

if you treat characters above 7f as if they were ordinary letters and make
no assumption of word-length or display width you should be fairly safe,

if you're hoping to identify digits and punctuation in unusual scripts
(Chinese, Sinhala, Sanscrit, Klingon etc) you'll need to do convert your
UTF-8 stream to unicode glyphs and pass them to the lexer.


Bye.
Jasen

 

[Douglas A. Gwyn]

I am using a lexer (lex specification supplied to lex) to parse data,
and one of the requirements is to handle UTF-8 characters. My
understanding is that the first non-ascii character's byte will be >
0x7f in a UTF-8 character If I look for the same in yytext -will that
suffice? Is there some std function that one can use to operate on the
input stream? I want my code to be locale agnostic.

You need to check that your version of "lex" supports wide characters,
which most do not. Otherwise you have to lex every possible character
into a token, which is almost certainly not what you want to do.

In most situations, it is easier to hand-code a lexer than to use "lex",
and here is a case where this is even more likely to be the case.

Convert the UTF-8 to 31-bit "Unicode" points and handle characters
solely as "wide" characters throughout.