std::locale question

[Brad Tilley]

I'm experimenting with std::locale. This code works:

int main()
{
// Get the run time locale.
const std::locale rt_locale( std::locale("") );

// Make the run time locale global.
std::locale::global( rt_locale );

// Print a few non-English Characters
std::wcout << L"\x2660 \x2661 \x2662 \x2663" << std::endl;

return 0;
}

However, adding this line breaks it:

// Print the run time locale
std::cout << rt_locale.name() << std::endl;

Here is working output:

$ ./a.out
♠ ♡ ♢ ♣

Here is broken output:

$ ./a.out
en_US.utf8
` a b c

I don't understand why .name() would cause this. I've never worked
with std::locale before, so I'm probably making some fundamental
mistake. Any advice is appreciated. Here's the g++ version I'm using:

g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5) 4.4.3

 

[Vaclav Haisman]

Brad Tilley wrote, On 2.3.2011 3:23:

I'm experimenting with std::locale. This code works:

int main()
{
// Get the run time locale.
const std::locale rt_locale( std::locale("") );

Why no just 'const std::locale rt_locale("")'?

// Make the run time locale global.
std::locale::global( rt_locale );

// Print a few non-English Characters
std::wcout << L"\x2660 \x2661 \x2662 \x2663" << std::endl;

return 0;
}

However, adding this line breaks it:

// Print the run time locale
std::cout << rt_locale.name() << std::endl;

Try imbuing both streams with the locale:
std::wcout.imbue (rt_locale);
std::cout.imbue (rt_locale);

 

[James Kanze]

I'm experimenting with std::locale. This code works:

int main()
{
// Get the run time locale.
const std::locale rt_locale( std::locale("") );

// Make the run time locale global.
std::locale::global( rt_locale );

// Print a few non-English Characters
std::wcout << L"\x2660 \x2661 \x2662 \x2663" << std::endl;

return 0;
}

However, adding this line breaks it:

// Print the run time locale
std::cout << rt_locale.name() << std::endl;

Here is working output:

$ ./a.out
♠ ♡ ♢ ♣

Here is broken output:

$ ./a.out
en_US.utf8
` a b c

I don't understand why .name() would cause this. I've never worked
with std::locale before, so I'm probably making some fundamental
mistake. Any advice is appreciated. Here's the g++ version I'm using:

g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5) 4.4.3

I'd say that the implementation is broken, but not in the way
you might think. The locale used by a stream is set when the
stream is constructed, and can only be change by imbue
afterwards. Both cout and wcout are required to be constructed
before main begins execution; in your case, at a time when the
global locale is the "C" locale (which normally doesn't support
UTF-8, although I think it would be allowed to). I suspect that
what is happening is that the locale of the various standard
streams isn't getting set until the first use of any one of
them: if the first output is after you have changed the global
locale, you get the new globale locale; if the first output is
before, you get the previous locale.

This is not legal, but it arguably makes sense to defer setting
the locale until the first use of a stream. (But even then, it
should only set the locale for that stream: outputting to cout
should in no case change the locale of wcout.)

Anyway, I'd suggest that when you change the global locale at
the start of the program, you also imbue the new locale in all
of the standard streams.

 

[Brad Tilley]

I'd say that the implementation is broken, but not in the way
you might think.  The locale used by a stream is set when the
stream is constructed, and can only be change by imbue
afterwards.  Both cout and wcout are required to be constructed
before main begins execution; in your case, at a time when the
global locale is the "C" locale (which normally doesn't support
UTF-8, although I think it would be allowed to).  I suspect that
what is happening is that the locale of the various standard
streams isn't getting set until the first use of any one of
them: if the first output is after you have changed the global
locale, you get the new globale locale; if the first output is
before, you get the previous locale.

This is not legal, but it arguably makes sense to defer setting
the locale until the first use of a stream.  (But even then, it
should only set the locale for that stream: outputting to cout
should in no case change the locale of wcout.)

Anyway, I'd suggest that when you change the global locale at
the start of the program, you also imbue the new locale in all
of the standard streams.

Thanks, I had tried imbue before posting the question here, it does
not work.

// Imbue
std::cout.imbue( rt_locale );
std::wcout.imbue( rt_locale );

I can recreate the problem on a different system with a slightly
different compiler:

g++ --version
g++ (Debian 4.4.5-8) 4.4.5

 

[Bart van Ingen Schenau]

I'm experimenting with std::locale. This code works:

int main()
{
// Get the run time locale.
const std::locale rt_locale( std::locale("") );

// Make the run time locale global.
std::locale::global( rt_locale );

// Print a few non-English Characters
std::wcout << L"\x2660 \x2661 \x2662 \x2663" <<
std::endl;

return 0;
}

However, adding this line breaks it:

// Print the run time locale
std::cout << rt_locale.name() << std::endl;

Here is working output:

$ ./a.out
♠ ♡ ♢ ♣

Here is broken output:

$ ./a.out
en_US.utf8
` a b c

I don't understand why .name() would cause this. I've never
worked with std::locale before, so I'm probably making some
fundamental mistake. Any advice is appreciated. Here's the g++
version I'm using:

g++ --version
g++ (Ubuntu 4.4.3-4ubuntu5) 4.4.3

Your problem does not lie with the use of locale::name, but with using
std::cout to print it.

std::cout and std::wcout use the same underlying stream (stdout) to produce
their output.
The problem is that stdout can not handle a mixture of narrow output (from
cout) and wide output (from wcout). The first use of a stream determines its
orientation (narrow or wide) and it keeps that orientation for the rest of
its lifetime, or until it is explicitly changed by the user (with a call to
fwide).

The best option is not to mix cout and wcout in a program.
The second-best option is to set the orientation of stdout explicitly when
changing between cout and wcout (or vice versa).

Bart v Ingen Schenau

 

[Brad Tilley]

Your problem does not lie with the use of locale::name, but with using
std::cout to print it.

Thank you! That fixed it. I had to covert the std::string returned by
locale::name to a std::wstring in order to use wcout. I suppose this
is how most people handle this.

// Print the run time locale
std::wcout << s_to_ws(rt_locale.name()) << std::endl;


inline const std::wstring s_to_ws( const std::string& s )
{
std::wstring ws;
ws.assign( s.begin(), s.end() );

return ws;
}

 

[James Kanze]

Your problem does not lie with the use of locale::name, but with using
std::cout to print it.

std::cout and std::wcout use the same underlying stream
(stdout) to produce their output.

They shouldn't. They're both synchronized with stdio, but
they're separate objects, with their own streambuf. And it is
the streambuf which should be handling any character
translation. (Of course, the standard doesn't require the
standard streams to have any particular type of streambuf, and
the character translation is defined by the derived streambuf
type, so I guess anything the implementation does is legal.)

The problem is that stdout can not handle a mixture of narrow
output (from cout) and wide output (from wcout). The first use
of a stream determines its orientation (narrow or wide) and it
keeps that orientation for the rest of its lifetime, or until
it is explicitly changed by the user (with a call to fwide).

That's true for stdio, but not for iostream.

There are restrictions with regards to changing encodings, I
think; once you've used a multibyte encoding, or maybe it's a
non-trivial encoding, you can't change. But you can read or
write a certain number of characters in "C" locale, then change
encodings. (This is necessary if you want to read XML, for
example.)

The best option is not to mix cout and wcout in a program.
The second-best option is to set the orientation of stdout
explicitly when changing between cout and wcout (or vice
versa).

Changes made in stdout do not, or at least should not, in a
standard conforming implementation, affect cout and wcout.

 

[Bart van Ingen Schenau]

They shouldn't.

I haven't checked the C++0x draft, but C++03 states:

[lib.narrow.stream.objects] 27.3.1/3: "The object cout controls output to a
stream buffer associated with the object stdout, declared in <cstdio>
(27.8.2)"
[lib.wide.stream.objects] 27.3.2/3: "The object wcout controls output to a
stream buffer associated with the object stdout, declared in <cstdio>
(27.8.2)"

Furthermore, there is this requirement:
[lib.iostream.objects] 27.3/2: "Mixing operations on corresponding wide- and
narrow-character streams follows the same semantics as mixing such
operations on FILEs, as specified in Amendmend 1 of the ISO C standard.
[...]"
The referenced requirement in the C standard states that such mixing is not
allowed without resetting the orientation of the stream.

I would expect wcout and cout to be two corresponding wide- and
narrow-character streams, even if they don't actually use stdout to produce
output on the standard output stream.

Changes made in stdout do not, or at least should not, in a
standard conforming implementation, affect cout and wcout.

Given the preceding quotes from the standard, I am not convinced you are
correct here.
A question (either on one of the C++ usenet groups or on stackoverflow)
triggered me to look into this issue with mixing cout and wcout in one
program.
The result was that, at least with GCC, calling fwide(stdout, X) when
switching between cout and wcout (or vice versa) made the output come out
correcly, while that did not happen without the fwide calls.
According to my understanding of the standard, that is also the expected
behaviour of a conforming implementation.

Bart v Ingen Schenau