main(int argc, wchar_t *argv[]) on linux i386 (redhat)

[interec]

Hi Folks,

I am writing a c++ program on redhat linux using main(int argc,
wchar_t *argv[]). $LANG on console is set to "en_US.UTF-8". g++
compiler version is 3.4.6.

Q1. what is the encoding of data that I get in argv[] ?

Q2. what is encoding of string constants defined in programs (for
example L"--count") ?

Q3. when I run the program as:

./a.out --count 1

Why does (wcscmp(argv[1], L"--count") == 0) always evaluate to
false?
What is the workaround. How do I make it evaluate to true?

Q4: when I do wstring

mystring = argv[1];

is mystring considered to be a UTF-8 string?


Q5: In case I use main(int argc, char** argv),

what is the encoding of characters in argv[] ?

Q6. What is the difference between main(int arc, wchar_t *argv[]) and
main(int argc, char *argv[]). Any document that describes this? My
compiler is gcc 3.4.6

Thanks

 

[Gianni Mariani]

Hi Folks,

I am writing a c++ program on redhat linux using main(int argc,
wchar_t *argv[]). $LANG on console is set to "en_US.UTF-8". g++
compiler version is 3.4.6.

main(int argc,wchar_t *argv[]) is not standard. It may compile and link
but it's probably not doing what you think it does.

Q1. what is the encoding of data that I get in argv[] ?

The encoding of the command line. It will probably be an 8bit /
multibyte encoding, dot a wide character (wchar_t) encoding.

Q2. what is encoding of string constants defined in programs (for
example L"--count") ?

This is a g++ question but in short, it depends on the compiler and the
encoding of the source file.

Q3. when I run the program as:

./a.out --count 1

Why does (wcscmp(argv[1], L"--count") == 0) always evaluate to
false?
What is the workaround. How do I make it evaluate to true?

Don't use what_t

Q4: when I do wstring

mystring = argv[1];

is mystring considered to be a UTF-8 string?

try it and see.

Q5: In case I use main(int argc, char** argv),

what is the encoding of characters in argv[] ?

The encoding the caller put in there.

Q6. What is the difference between main(int arc, wchar_t *argv[]) and
main(int argc, char *argv[]). Any document that describes this? My
compiler is gcc 3.4.6

Check with the GCC docs, but I don't think main(int arc, wchar_t
*argv[]) is supported at all.

 

[SasQ]

Dnia Fri, 23 Mar 2007 14:54:52 -0700, interec napisa³(a):

I am writing a c++ program on redhat linux using
main(int argc, wchar_t *argv[]).

C++ Standard specifies only those two signatures for main:

int main();
int main(int argc, char** argv); //or char* argv[], whatever

Other signatures are allowed only in embedded environments.

$LANG on console is set to "en_US.UTF-8".

Console character encoding doesn't matter.

Q1. what is the encoding of data that I get in argv[] ?

Probably the one used for command line.

Q2. what is encoding of string constants defined in
programs (for example L"--count") ?

The encoding of string literal constants isn't precisely
defined and in most cases it is mapped from source character
set to machine code directly, without changes.

Q3. when I run the program as:

./a.out --count 1

Why does (wcscmp(argv[1], L"--count") == 0) always
evaluate to false?

Because L"--count" is composed from wide characters [stored
in more bytes than a sizeof(char) and of 'wchar_t' type].
It is encoded in most cases using Unicode UTF-16. So, unlike
your console, it is not UTF-8].

What is the workaround. How do I make it evaluate to true?

Use standard signature:

int main(int argc, char* argv[]);

and interpret argv[1] as UTF-8. Characters in UTF-8 encoded
strings are plain one-byte 'char's but the international
characters are encoded as more-than-one-byte sequences.
If you program for Unix-like system, you may use iconv library
to transcode strings from/to multiple character sets.

Q4: when I do

wstring mystring = argv[1];

is mystring considered to be a UTF-8 string?

No. It is considered to be wide-character string.
UTF-8 is not a wide-character string, it's normal
one-byte-character string, but some characters are
encoded as multiple bytes.

Q5: In case I use main(int argc, char** argv),
what is the encoding of characters in argv[] ?

The same as in the command line.

Q6. What is the difference between

main(int arc, wchar_t *argv[])

and
main(int argc, char *argv[]).

The former is not standard.

Any document that describes this?

ANSI/ISO/IEC Standard 14882 - The C++ Programming Language

 

[interec]

Ok, here is a sample program. $LANG is set to "en_US.UTF-8" on my
console. When I invoke this program from command line as:

$ ./a.out "आ१२३४५६"

argument argv[1] gets set to an array of multibyte characters in UTF-8
format. Correct? Then as a test, I convert argv[1] to a wstring as
follows:

wstring sSql = make_wide(argv[1]);

and finally I print it out back to the console as:

wcout << sSql << endl;

The problem is that nothing is getting printed out on the console. I
tried using locale, but that didn't help either. Could someone please
explain whats going on and how I can print out wstring here. Full
sample code is given below. My compiler is g++ 3.4.6. OS is red hat
linux 4ES.

If I invoke the same program using english characters, everything
works as expected:

$ ./a.out "select"
select

The problem occurs only when I enter multi-byte characters as command
line args.

------------------------------------------------ SAMPLE PROGRAM
------------------------------------------------
#include <iostream>
#include <iomanip>
#include <locale>

using namespace std;

string make_narrow(const wstring sArg);
wstring make_wide(const string sArg);

string make_narrow(const wstring sArg)
{
std::string result;
std::locale loc;
for(unsigned int i= 0; i < sArg.size(); ++i)
{

(loc).narrow(sArg, 0);

}
return result;
}

wstring make_wide(const string sArg)
{
std::wstring result;
std::locale loc;
for(unsigned int i= 0; i < sArg.size(); ++i)
{

(loc).widen(sArg);

}
return result;
}

/**
* main is supposed to take command line argument string argv[1] as
* input and print it out back on console in std::wstring form.
*
* $LANG is set to "en_US.UTF-8" on console.
*
* This function works with argv[1] contains only english characters.
* Function does not work with argv[1] contains a string containing
* multibyte characters.
*
*/
int main(int argc, char *argv[])
{
locale aLoc("en_US.UTF-8");
locale::global(aLoc);
wcout.imbue(aLoc);

wstring sSql = make_wide(argv[1]);

wcout << sSql << endl;
}

----------------------------------------------------------------------------------------------------------------

Dnia Fri, 23 Mar 2007 14:54:52 -0700, interec napisa³(a):

I am writing a c++ program on redhat linux using
main(int argc, wchar_t *argv[]).

C++ Standard specifies only those two signatures for main:

int main();
int main(int argc, char** argv); //or char* argv[], whatever

Other signatures are allowed only in embedded environments.

$LANG on console is set to "en_US.UTF-8".

Console character encoding doesn't matter.

Q1. what is the encoding of data that I get in argv[] ?

Probably the one used for command line.

Q2. what is encoding of string constants defined in
programs (for example L"--count") ?

The encoding of string literal constants isn't precisely
defined and in most cases it is mapped from source character
set to machine code directly, without changes.

Q3. when I run the program as:

./a.out --count 1

Why does (wcscmp(argv[1], L"--count") == 0) always
evaluate to false?

Because L"--count" is composed from wide characters [stored
in more bytes than a sizeof(char) and of 'wchar_t' type].
It is encoded in most cases using Unicode UTF-16. So, unlike
your console, it is not UTF-8].

What is the workaround. How do I make it evaluate to true?

Use standard signature:

int main(int argc, char* argv[]);

and interpret argv[1] as UTF-8. Characters in UTF-8 encoded
strings are plain one-byte 'char's but the international
characters are encoded as more-than-one-byte sequences.
If you program for Unix-like system, you may use iconv library
to transcode strings from/to multiple character sets.

Q4: when I do

wstring mystring = argv[1];

is mystring considered to be a UTF-8 string?

No. It is considered to be wide-character string.
UTF-8 is not a wide-character string, it's normal
one-byte-character string, but some characters are
encoded as multiple bytes.

Q5: In case I use main(int argc, char** argv),
what is the encoding of characters in argv[] ?

The same as in the command line.

Q6. What is the difference between

main(int arc, wchar_t *argv[])

and
main(int argc, char *argv[]).

The former is not standard.

Any document that describes this?

ANSI/ISO/IEC Standard 14882 - The C++ Programming Language

 

[angrybaldguy]

Ok, here is a sample program. $LANG is set to "en_US.UTF-8" on my
console. When I invoke this program from command line as:

$ ./a.out "आ१२३४५६"

argument argv[1] gets set to an array of multibyte characters in UTF-8
format. Correct?

The behaviour you're asking about is not defined by the C++ language.
You may wish to investigate, eg., comp.unix.programmer, or your
distribution's documentation about how it launches programs.

Then as a test, I convert argv[1] to a wstring as follows:

wstring sSql = make_wide(argv[1]);

and finally I print it out back to the console as:

wcout << sSql << endl;

The problem is that nothing is getting printed out on the console. I
tried using locale, but that didn't help either.

Have you tried examining the actual bytes being received in the argv
entries? Here's a short program (C++) that prints the character, hex
value, and decimal value of each character of each argv entry; try
running it with the above command line and seeing what it prints.

// Begin args.cpp
#include <iostream>
#include <iomanip>

void hexDump (const char *data, std:stream &out) {
using std::endl;
using std::hex;
using std::dec;

for (int character = 0; data[character]; ++character) {
int asInt = data[character];

out << "Byte " << character << ": ";
out << hex << asInt;
out << " (" << dec << asInt << "): ";
out << data[character] << endl;
}
}

int main (int argc, char *argv[]) {
using std::cout;
using std::endl;

for (int argIndex = 0; argIndex < argc; ++argIndex) {
const char *argument = argv[argIndex];

cout << "Argument " << argIndex << ":" << endl;
hexDump (argument, cout);
}
}
// End of args.cpp

Then at least you will be aware of the encoded representation of the
command-line arguments. You may need to dike out out <<
data[character] part if the encoding is multibyte and any of the
individual bytes are interpreted as control characters; alternately,
<ot>pipe the output through less(1), which sanitizes its input before

#include <iostream>
#include <iomanip>
#include <locale>

using namespace std;

You should probably pick one of "using namespace std;" and using fully-
qualified names. Below you do both, whcih is somewhat confusing.

string make_narrow(const wstring sArg);
wstring make_wide(const string sArg);

string make_narrow(const wstring sArg)
{
std::string result;
std::locale loc;
// ...

Finally, please trim quoted text and reply inline, rather than quoting
the entire thread yet one more time and replying at the top. The
former is the accepted style for this newsgroup and the latter reads
"backwards".