how to convert characters to upper case in utf8 env.

[csanjith]

Hi, i have a situaion where i need to convert the characters entered in

an text field to upper case using C. The configuration id utf8
environment in which user can enter any character (single , double,
triple byte etc). I need to convert to upper case only those characters

which has got upper case. ie if an user enter bot english and japanese
characters in the text field, then I should convert only english
characters, not japanese.

I have seen that the C functions toupper() and tolower() handles multi
byte characters from Solaris 8. I am not sure with other platforms.


can any one suggest the best approch for the above scenario.

 

[Micah Cowan]

Hi, i have a situaion where i need to convert the characters entered in

Hi. Do you know you triple-posted this?

an text field to upper case using C. The configuration id utf8
environment in which user can enter any character (single , double,
triple byte etc). I need to convert to upper case only those characters

which has got upper case. ie if an user enter bot english and japanese
characters in the text field, then I should convert only english
characters, not japanese.

I have seen that the C functions toupper() and tolower() handles multi
byte characters from Solaris 8. I am not sure with other platforms.

can any one suggest the best approch for the above scenario.

The encodings supported by your C implementation for operations in
toupper() and tolower() are implementation-defined: you'll need to
look it up in your documentation.

You might need to use setlocale() with something like "en_US.UTF8",
and restore the locale afterwards. This may or may not work. And you
still won't be able to work with multibyte strings directly: you'll
have to convert to and from wchar_t's.

Your best bet is to use a (off-topic) specialized library devoted to
manipulating UTF8 strings. IBM has one:
http://www-306.ibm.com/software/globalization/icu/index.jsp

This provides a u_strToLower() function in ustring.h. Please don't
post here for questions regarding this library, however, as it is
off-topic for this NG.

HTH,
-Micah

 

[loufoque]

Micah Cowan a écrit :

Your best bet is to use a (off-topic) specialized library devoted to
manipulating UTF8 strings. IBM has one:

And glib can do it too.

 

[Vladimir S. Oka]

Hi. Do you know you triple-posted this?

It's the blinkin' Google. It does it sometimes.

--
BR, Vladimir

"There is hopeful symbolism in the fact that flags do not wave in a
vacuum."
-- Arthur C. Clarke

 

[websnarf]

Hi, i have a situaion where i need to convert the characters entered in

an text field to upper case using C. The configuration id utf8
environment in which user can enter any character (single , double,
triple byte etc). I need to convert to upper case only those characters

Latin based uppercasing is easy, just convert to those characters
exactly in the lower-case or upper-case ASCII range. This is one of
the properties of UTF-8. However, to perform correct case change over
the whole Unicode range, you need to simply know which characters have
either a upper case or capitalization case alternative character (as
well as the reverse.) This information is available from the standard
Unicode data table.

Oh yeah, and this off topic here in comp.lang.c. ANSI C does not have
a notion of portable internationalization, let alone Unicode (though
some compilers implement wchar_t as Unicode, this cannot be relied
upon.)

 

[William Ahern]

Hi, i have a situaion where i need to convert the characters entered in

an text field to upper case using C. The configuration id utf8
environment in which user can enter any character (single , double,
triple byte etc). I need to convert to upper case only those characters

which has got upper case. ie if an user enter bot english and japanese
characters in the text field, then I should convert only english
characters, not japanese.

I have seen that the C functions toupper() and tolower() handles multi
byte characters from Solaris 8. I am not sure with other platforms.

It would seem improbable that toupper(), on Solaris or elsewhere, could
give the correct output for all valid input when using UTF-8, UTF-16,
UTF-32 or any other Unicode encoding variant.

For all Unicode encoding variants there are some "characters" (or
"graphemes" in Unicode terminology) which can be encoded
equivalently across multiple sets of integer values. Assume UTF-32, and
our Unicode string is an array of uint32_t integers. The Latin-1 character
a+umlaut could be described equivalently within the range of one 32-bit
integer with a value of 0x000000C1 or by combining two integers,
0x00000041-0x00000301. The latter representation could not be passed to
toupper() or tolower(), considering that neither can take an array.

[http://www.unicode.org/faq/char_combmark.html#8]

Hmmmm. Do any Unicode gurus know if 0x0061-0301 would accomplish a capital
A + umlaut? Regardless, I strongly suspect that there are many graphemes
in many scripts where such a trick could never work, but where notions
like uppercase or lowercase are still meaningful.

 

[William Ahern]

Latin based uppercasing is easy, just convert to those characters
exactly in the lower-case or upper-case ASCII range. This is one of
the properties of UTF-8. However, to perform correct case change over
the whole Unicode range, you need to simply know which characters have
either a upper case or capitalization case alternative character (as
well as the reverse.) This information is available from the standard
Unicode data table.

Latin != ASCII. ASCII is 7-bit, ISO Latin encodings are 8-bit. Non-ASCII
code points, in UTF-8, are multibyte. How do you pass an array to toupper()?

Oh yeah, and this off topic here in comp.lang.c. ANSI C does not have
a notion of portable internationalization, let alone Unicode (though
some compilers implement wchar_t as Unicode, this cannot be relied
upon.)

The wide-character API is not sufficient to support Unicode.
But you're right, this is off-topic. Anybody know where this would be
on-topic, though?

 

[Richard Bos]

For all Unicode encoding variants there are some "characters" (or
"graphemes" in Unicode terminology) which can be encoded
equivalently across multiple sets of integer values. Assume UTF-32, and
our Unicode string is an array of uint32_t integers. The Latin-1 character
a+umlaut could be described equivalently within the range of one 32-bit
integer with a value of 0x000000C1 or by combining two integers,
0x00000041-0x00000301. The latter representation could not be passed to
toupper() or tolower(), considering that neither can take an array.

[http://www.unicode.org/faq/char_combmark.html#8]

Hmmmm. Do any Unicode gurus know if 0x0061-0301 would accomplish a capital
A + umlaut?

If I read the Unicode Standard correctly, yes, it would. However, the
right question is: do you really _want_ to capitalise an accented lowed
case letter to an accented upper case letter? In Dutch you wouldn't.

There are (at least) two reasonable C solutions:
- trust that your implementation handles this correctly, for example by
letting the sysadmin of the system the program runs on install
language-specific libraries for the <ctype.h> functions, and just use
tolower() and toupper(), as you would otherwise;
- assume that you know better than J. Random Sysadmin which characters
you want to capitalise, and write your own case-changing functions
with knowledge about the Unicode tables.
Something is to be said for either solution; the first is simpler and
more flexible, in the second case the results are more strictly known.

Richard

 

[Eric Sosman]

Hi, i have a situaion where i need to convert the characters entered in

an text field to upper case using C. The configuration id utf8
environment in which user can enter any character (single , double,
triple byte etc). I need to convert to upper case only those characters

Latin based uppercasing is easy, just convert to those characters
exactly in the lower-case or upper-case ASCII range. [...]

Note that all of "àáâãäåæçèéêëìíîïðñòóôõöøùúûüý" appear
in the alphabets of Latinic (Latinous?) languages, are
outside the ASCII lower-case range, and yet have upper-
case equivalents.