Question about language setting

D

Dave Saville

I have a perl script that checks a program's "data base" for
consistancy.

A German user said it would not run and threw a "Feature bundle 5.0.0"
not supported from feature.pm. This was because I call Carp.

He now says

<qoute>
the perl problem seems to be solved: it's not a problem with
installation or version. The problem is the setting for "LANG": It has
to be LANG=en_US.UTF-8 and not LANG=de_DE_EURO which is the standard
for german!
</quote>

Surely (base) perl modules aren't dependent on language?
 
R

Rainer Weikusat

Dave Saville said:
A German user said it would not run and threw a "Feature bundle 5.0.0"
not supported from feature.pm. This was because I call Carp.

He now says

<qoute>
the perl problem seems to be solved: it's not a problem with
installation or version. The problem is the setting for "LANG": It has
to be LANG=en_US.UTF-8 and not LANG=de_DE_EURO which is the standard
for german!
</quote>

Surely (base) perl modules aren't dependent on language?

According to a completely random web search result,

https://www.illumos.org/issues/1203

the issue is possibly that the 5.0 becomes 5,0 when formatted
according to German conventions. The thread and related bug & bugfix has
some more information on this (AFAIK, 'feature bundle 5.0.0' means
'Perl5').
 
D

Dave Saville

According to a completely random web search result,

https://www.illumos.org/issues/1203

the issue is possibly that the 5.0 becomes 5,0 when formatted
according to German conventions. The thread and related bug & bugfix has
some more information on this (AFAIK, 'feature bundle 5.0.0' means
'Perl5').

Well that seems to suggest that it is not perl but something else

<qoute>
root source found, the bug is not in perl, its in Illumos locales,
</quote>

Whatever that is. Surely perl must be used in non English speaking
countries? :)
 
R

Rainer Weikusat

Dave Saville said:
Well that seems to suggest that it is not perl but something else

<qoute>
root source found, the bug is not in perl, its in Illumos locales,
</quote>

Whatever that is. Surely perl must be used in non English speaking
countries? :)

As I already wrote: The thread and the associated OpenSlowalrus bug &
bug fix provide some more background information on this. In particular,
the suggested workaround (set LC_NUMERIC to a 'decimal point' locale)
should be sufficient to determine if this is really the same
problem. Depending on how you feel about that, you might also just tell
your user that "Locale support on your system is obviously broken, get
of my lawn, sucker!" -- the open source way of dealing with
'uninteresting' user problems ...
 
P

Peter J. Holzer

I have a perl script that checks a program's "data base" for
consistancy.

A German user said it would not run and threw a "Feature bundle 5.0.0"
not supported from feature.pm. This was because I call Carp.

It would be useful to know where this error occurs. Just "a perl script"
is a bit vague.

I get this error if I write something like

use feature ':5.0.0';

but that is independent of the locale setting. Feature.pm was introduced
in 5.10 and there aren't any feature bundles less than :5.10.

Also, details about the system in question (OS/distribution, perl
version) would also be useful.
He now says

<qoute>
the perl problem seems to be solved: it's not a problem with
installation or version. The problem is the setting for "LANG": It has
to be LANG=en_US.UTF-8 and not LANG=de_DE_EURO which is the standard
for german!
</quote>

Surely (base) perl modules aren't dependent on language?

They shouldn't be. It might be a bug in perl, in the module or in a
system library. Or it might be a bug in your script. It is impossible to
tell with the information you have given.

hp
 
R

Rainer Weikusat

Peter J. Holzer said:
[...]
He now says

<qoute>
the perl problem seems to be solved: it's not a problem with
installation or version. The problem is the setting for "LANG": It has
to be LANG=en_US.UTF-8 and not LANG=de_DE_EURO which is the standard
for german!
</quote>

Surely (base) perl modules aren't dependent on language?

They shouldn't be. It might be a bug in perl, in the module or in a
system library. Or it might be a bug in your script. It is impossible to
tell with the information you have given.

Considering the statement from 'the user', this must be something which
causes the behaviour of some perl code to change because it is using a
German locale. Considering what is in a locale definition, this makes
LC_NUMERIC the most likely candidate, especially considering that this
is the only locale-information used as part of version checking (by
Perl_upg_version in util.c).
 
P

Peter J. Holzer

Considering the statement from 'the user', this must be something which
causes the behaviour of some perl code to change because it is using a
German locale. Considering what is in a locale definition, this makes
LC_NUMERIC the most likely candidate, especially considering that this
is the only locale-information used as part of version checking (by
Perl_upg_version in util.c).

Yes, I know. But I couldn't reproduce it. And that's not very
surprising: Many people use perl with a German locale, so it can't be
something generic like "perl doesn't work with a German locale" or "use
feature doesn't work with a German locale". It has to be something more
specific, like a bug in the locale implementation of the user's system
(but Dave hasn't told us what system it is) or the way a certain feature
is used (but Dave hasn't told us anything about his script). But it is
impossible to find out what it is unless Dave volunteers more
information.

hp
 
D

Dave Saville

Yes, I know. But I couldn't reproduce it. And that's not very
surprising: Many people use perl with a German locale, so it can't be
something generic like "perl doesn't work with a German locale" or "use
feature doesn't work with a German locale". It has to be something more
specific, like a bug in the locale implementation of the user's system
(but Dave hasn't told us what system it is) or the way a certain feature
is used (but Dave hasn't told us anything about his script). But it is
impossible to find out what it is unless Dave volunteers more
information.

A little difficult to post extra info from the user as I only have a
jpeg of the errors rather than something I can cut and paste :-(

The script starts:

use strict;
use warnings;
use Carp:
use Encode;

Compilation failed in require at .... line 3 - ie the "use Carp" line.

System is OS/2 his perl is 5.10.0. (Yes I know :) )

If you follow the links from the posted forum you end up with a C test
program. I have built that on my system but as I am in the UK my
system is defaulting to using a 'point' anyway. I have yet to persuade
it to start up in a different locale. I know how to get a program to
use a different language but that does not seem to help. I have posted
this problem to an OS/2 tech group in the hope they may throw some
light on the problem - assuming it is a similar one in that
setlocale() is not working correctly as per the forum post.
 
R

Rainer Weikusat

Dave Saville said:
[...]
[...]
Yes, I know. But I couldn't reproduce it. And that's not very
surprising: Many people use perl with a German locale, so it can't be
something generic like "perl doesn't work with a German locale" or "use
feature doesn't work with a German locale". It has to be something more
specific, like a bug in the locale implementation of the user's system
[...]

A little difficult to post extra info from the user as I only have a
jpeg of the errors rather than something I can cut and paste :-(

The script starts:

use strict;
use warnings;
use Carp:
use Encode;

Compilation failed in require at .... line 3 - ie the "use Carp" line.

System is OS/2 his perl is 5.10.0. (Yes I know :) )

If you follow the links from the posted forum you end up with a C test
program. I have built that on my system but as I am in the UK my
system is defaulting to using a 'point' anyway. I have yet to persuade
it to start up in a different locale.

According to some more searching around on the web, the OS/2 command to
set a locale is (example)

set lang=de_DE
[http://www.borgendale.com/locale.htm]

Results as to whether or not setlocale works on OS/2 are somewhat
unclear. Assuming your system starts with 'English' language settings,
you could use a modified test program,

-----------
#include <locale.h>
#include <stdio.h>

int main(void) {
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "de_DE");
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "C");
printf("%f\n",2.5);

return 0;
}
-----------

this should print (possibly with variations in the number of trailing
zeroes)

2.500000
2,500000
2.500000

If it doesn't, in particular, if there's a dot instead of a comma in the
second line, setlocale doesn't work as it is supposed to and if you
can't switch from English to German in this way, chances are very high
that switching from German to English also won't work.
 
D

Dave Saville

If it doesn't, in particular, if there's a dot instead of a comma in the
second line, setlocale doesn't work as it is supposed to and if you
can't switch from English to German in this way, chances are very high
that switching from German to English also won't work.

It would appear we have the same or very similar problem with
setlocale() as reported upthread. It is being looked into. FWIW I now
have the original error in text form:

Feature bundle "5.0.0" is not supported by Perl 5.10.0 at
D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 207
feature::croak('Feature bundle "5.0.0" is not supported by Perl
5.10.0') called at D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 201
feature::unknown_feature_bundle(5.0.0) called at
D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 152
feature::import('feature', ':5.0.0') called at
D:/PROGRAMS/PERL/LIB/5.10.0/Exporter.pm line 3
Exporter::BEGIN() called at D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm
line 0
eval {...} called at D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 0
require Exporter.pm called at D:/PROGRAMS/PERL/LIB/5.10.0/Carp.pm
line 14
require Carp.pm called at c:\progr\pmmail3\bin\pmmail_folder_check.pl
line 13
main::BEGIN() called at D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 0
eval {...} called at D:/PROGRAMS/PERL/LIB/5.10.0/feature.pm line 0
BEGIN failed--compilation aborted.
Compilation failed in require at D:/PROGRAMS/PERL/LIB/5.10.0/Carp.pm
line 14.
Compilation failed in require at
c:\progr\pmmail3\bin\pmmail_folder_check.pl line 13.
BEGIN failed--compilation aborted at
c:\progr\pmmail3\bin\pmmail_folder_check.pl line 13.

The only difference between this and the stripped down code I posted
earlier is that this script starts with ten comment lines so 13 above
is my test's 3. ie the use Carp; line.
 
M

Martin Gieretz

Am 24.12.2013 11:39, schrieb Dave Saville:
A little difficult to post extra info from the user as I only have a
jpeg of the errors rather than something I can cut and paste :-(

The script starts:

use strict;
use warnings;
use Carp:
use Encode;

Compilation failed in require at .... line 3 - ie the "use Carp" line.

System is OS/2 his perl is 5.10.0. (Yes I know :) )

If you follow the links from the posted forum you end up with a C test
program. I have built that on my system but as I am in the UK my
system is defaulting to using a 'point' anyway. I have yet to persuade
it to start up in a different locale. I know how to get a program to
use a different language but that does not seem to help. I have posted
this problem to an OS/2 tech group in the hope they may throw some
light on the problem - assuming it is a similar one in that
setlocale() is not working correctly as per the forum post.

i have not read the whole tread,
but how about using 'use Carp;' instead of 'use Carp:'?
^

bye
M
 
D

Dave Saville

On Fri, 27 Dec 2013 14:59:11 UTC, Martin Gieretz

i have not read the whole tread,
but how about using 'use Carp;' instead of 'use Carp:'?
^

Which would throw a syntax error not what we are seeing - My bad I
just typed the three "use" lines into the mail, not copy/paste.
 
D

Dave Saville

Dave Saville said:
He now says

<qoute>
the perl problem seems to be solved: it's not a problem with
installation or version. The problem is the setting for "LANG": It has
to be LANG=en_US.UTF-8 and not LANG=de_DE_EURO which is the standard
for german!
</quote>
[...]
Considering the statement from 'the user', this must be something which
causes the behaviour of some perl code to change because it is using a
German locale. Considering what is in a locale definition, this makes
LC_NUMERIC the most likely candidate,
[...]
Yes, I know. But I couldn't reproduce it. And that's not very
surprising: Many people use perl with a German locale, so it can't be
something generic like "perl doesn't work with a German locale" or "use
feature doesn't work with a German locale". It has to be something more
specific, like a bug in the locale implementation of the user's system
[...]

A little difficult to post extra info from the user as I only have a
jpeg of the errors rather than something I can cut and paste :-(

The script starts:

use strict;
use warnings;
use Carp:
use Encode;

Compilation failed in require at .... line 3 - ie the "use Carp" line.

System is OS/2 his perl is 5.10.0. (Yes I know :) )

If you follow the links from the posted forum you end up with a C test
program. I have built that on my system but as I am in the UK my
system is defaulting to using a 'point' anyway. I have yet to persuade
it to start up in a different locale.

According to some more searching around on the web, the OS/2 command to
set a locale is (example)

set lang=de_DE
[http://www.borgendale.com/locale.htm]

Results as to whether or not setlocale works on OS/2 are somewhat
unclear. Assuming your system starts with 'English' language settings,
you could use a modified test program,

-----------
#include <locale.h>
#include <stdio.h>

int main(void) {
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "de_DE");
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "C");
printf("%f\n",2.5);

return 0;
}
-----------

this should print (possibly with variations in the number of trailing
zeroes)

2.500000
2,500000
2.500000

If it doesn't, in particular, if there's a dot instead of a comma in the
second line, setlocale doesn't work as it is supposed to and if you
can't switch from English to German in this way, chances are very high
that switching from German to English also won't work.

Interestingly:

use strict;
use warnings;
use Carp;

printf("%f\n", 2.5);

Using perl 5.16.0

[T:\tmp]try.pl
Invalid version format (non-numeric data) at
u:/perl5/lib/5.16.0/Carp.pm line 3.

BEGIN failed--compilation aborted at u:/perl5/lib/5.16.0/Carp.pm line
3.
Compilation failed in require at try.pl line 3.
BEGIN failed--compilation aborted at try.pl line 3.

Carp line 3 says { use 5.006; }

Using perl 5.8.2

[T:\tmp]try.pl
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LC_ALL = (unset),
LANG = "de_DE_EURO"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
2.500000

Which is a darn site more useful IMHO.

As the OS/2 setlocale() seems to suffer the same problem as the other
OS above is there a perlish way around this one?

The problem I have is that I am comparing strings from two sources.
One where the string is in the local code page and the other in utf8.
I solved this by using Encode and friends but that introduces the
requirement for Carp and the above error :-(
 
P

Peter J. Holzer

Interestingly:

use strict;
use warnings;
use Carp;

printf("%f\n", 2.5);

Using perl 5.16.0

[T:\tmp]try.pl
Invalid version format (non-numeric data) at
u:/perl5/lib/5.16.0/Carp.pm line 3.

BEGIN failed--compilation aborted at u:/perl5/lib/5.16.0/Carp.pm line
3.
Compilation failed in require at try.pl line 3.
BEGIN failed--compilation aborted at try.pl line 3.

Carp line 3 says { use 5.006; }

Yup. I think somebody (Rainer?) had already pointed out this line as the
likely culprit.
Using perl 5.8.2

[T:\tmp]try.pl
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LC_ALL = (unset),
LANG = "de_DE_EURO"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
2.500000

Which is a darn site more useful IMHO.

As the OS/2 setlocale() seems to suffer the same problem as the other
OS above is there a perlish way around this one?

You could try

BEGIN { $ENV{LANG} = 'C' }

The problem I have is that I am comparing strings from two sources.
One where the string is in the local code page and the other in utf8.
I solved this by using Encode

Yes, Encode contains the necessary functions (But depending on the
source it may be more convenient to use an I/O filter or soemthing
similar instead of calling Encode::decode() explicitely).

and friends but that introduces the
requirement for Carp and the above error :-(

I'm not sure why using Encode requires the use of Carp, but in any case

{ use 5.006; }

is valid Perl and must compile successfully, regardless of any locale
settings.

Is OS/2 still officially supported by Perl? It isn't on
http://www.cpan.org/ports/ anymore.

A viable workaround might be to compile perl on OS/2 without locale
support (since it doesn't seem to work correctly anyway). But that would
also mean that your users need to use this locale-less perl binary,
which may break some other scripts they use.

hp
 
D

Dave Saville

Hi Ben
Since your broken locales are causing feature.pm to fail, you get weird
failures if it gets loaded, but not otherwise. Carp loads feature.pm
(unintentionally) on 5.16 or later, but not on earlier versions.

A clever man (ie not me) has found the problem.

perl is built with locale.h
libc is built with unidef.h

The definitions of LC_NUMERIC and LC_MONETARY are reversed between
the two. Correcting locale.h to match unidef.h results in the posted C
testcase to work as expected. Our perl porter is rebuilding a version
with this change.
It might be worth reporting a bug about this: broken locales are Not
Good, but they shouldn't cause perl to fail unless you explicitly try to
invoke locale functions.

Would you mind doing this please? I am not sure I can write a ticket
that others could act on.
 
D

Dave Saville

Now the first problem is solved, see elseehere in this tree, on to the
next :)

I have two input files the creation of which I have no control over.
One has been written, from C, using whatever codepage the user has.
The other file is encoded utf8.

Both files contain a file name. One of the consistancy checks is to
see if they match. I can read in the second file with open my $FILE2,
'<:encoding(utf8)", .......

So how to open the first?

open my $FILE1, '<', .....
open my $FILE1, <:raw', ....
open my $FILE1, '<', .........
binmode $FILE;

All seem to do exactly the same thing - except for \r\n processing.

or

open my $FILE1, '<:encoding(whatever)', ......

When perl converts to internal utf8 how does it decide to map code
points > 127 when it is not told the cp? It does not default to
current as I run with 850 and opening '<:encoding(cp850)' gives a
different result for points > 127 than does open '<'. It's complicated
by the fact that the field separator on the first file is hex DE that
will get mangled in the former case. But my $sep = decode("cp$cp",
"\xDE"); where $cp is the codepage seems to get around that one.

What I need to be able to do is

if ( $file1name eq $file2name ) where the names have accented
characters.

TIA
 
D

Dave Saville

It reads bytes and maps them directly to Unicode codepoints; which,
because of the way Unicode is defined, means it's effectively using
ISO8859-1.


No. Perl doesn't know anything about OS/2 codepages. (It could, perhaps,
be taught; that is, it would be possible to write a :codepage layer
which looked up the current codepage and used that; but it would be
highly unportable to do something like that unless it was explicitly
requested.)


When dealing with binary separators it's better to handle the data as
binary first, then decode to Unicode afterwards. That is, open the file
in :raw mode, split into fields on /\xDE/, and only then decode each
field from CP850 with decode().

What you were doing was effectively treating the file as 'fields
separated by I-grave' rather than 'fields separated by binary 0xDE'.
Those things happen to come out looking the same in a CP850 file,
because CP850 is a 1-byte encoding and 0xDE is a byte with defined
semantics, but in general you can't assume binary separators can be
meaningfully decoded from any given encoding.

Many thanks Ben. As I said I didn't "design" the 0xDE bit :) This
code is checking files of a program that only runs on OS/2 so
portability is not a question.
 
P

Peter J. Holzer

No. Perl doesn't know anything about OS/2 codepages. (It could, perhaps,
be taught; that is, it would be possible to write a :codepage layer
which looked up the current codepage and used that; but it would be
highly unportable to do something like that unless it was explicitly
requested.)

There is the Encode::Locale module, which allows you to do something
like
open(my $fh, '<:encoding(locale)', $filename);

I don't think it works on OS/2 (A colleague recently told me it doesn't
even work on Windows), but I think it would be better to extend an
existing module to work on more platforms than to add another
platform-specific module.

hp
 
B

Bjoern Hoehrmann

* Peter J. Holzer wrote in comp.lang.perl.misc:
There is the Encode::Locale module, which allows you to do something
like
open(my $fh, '<:encoding(locale)', $filename);

I don't think it works on OS/2 (A colleague recently told me it doesn't
even work on Windows), but I think it would be better to extend an
existing module to work on more platforms than to add another
platform-specific module.

http://www.nntp.perl.org/group/perl.i18n/2008/07/msg214.html has some
thoughts about Windows support in that area.
 
K

Keith Thompson

Rainer Weikusat said:
Results as to whether or not setlocale works on OS/2 are somewhat
unclear. Assuming your system starts with 'English' language settings,
you could use a modified test program,

-----------
#include <locale.h>
#include <stdio.h>

int main(void) {
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "de_DE");
printf("%f\n",2.5);
setlocale(LC_NUMERIC, "C");
printf("%f\n",2.5);

return 0;
}
-----------

this should print (possibly with variations in the number of trailing
zeroes)

2.500000
2,500000
2.500000

If it doesn't, in particular, if there's a dot instead of a comma in the
second line, setlocale doesn't work as it is supposed to and if you
can't switch from English to German in this way, chances are very high
that switching from German to English also won't work.

setlocale() returns a char* that's either a pointer to a string (if it
succeeded), or a null pointer (if it failed). It's probably worth
adding code to check for this rather than just depending on the output
of printf to change. (On my system, the first setlocale() call fails
because I don't have any German locales installed.)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,904
Latest member
HealthyVisionsCBDPrice

Latest Threads

Top