ambiguous results from pack

[Ralph]

I'm trying to do development on a Linux laptop to port to Solaris.
However, I've run into a pack command that's returning incorrect
results on my laptop.

I'm using Perl 5.8.0 on all platforms. I've included the output of
perl -V also below because the problem seems to be so low level.

Here's the command...

perl -e 'print pack("I", 149);'|od -x

The correct response, which I get on most platforms, is this...

00000000 0095 0000
00000004

However, on my laptop, where it really matters, I'm getting this...

00000000 95C2 0000
00000004

I'd write this off as some weird endian problem, except that I'm
getting the incorrect results on an i386 based machine, and getting
correct results on 2 other i386 based machines, one Linux and one
Cygwin.

Anyway, below is the output of perl -V. The first listing is the one
that's returning incorrect results, the rest are correct. I've
changed the hostname portion of the uname section, but otherwise these
are verbatim.

Any help at all would be appreciated. If necessary I'll recompile
Perl on the laptop with the config parameters that'll correct the
problem (if I can find out what those are).

Thanks for your time.

Ralph

=======================================================================
Laptop Linux returning incorrect results.
=======================================================================
Summary of my perl5 (revision 5.0 version 8 subversion 0)
configuration:
Platform:
osname=linux, osvers=2.4.20-6, archname=i686-linux
uname='linux laptop.somedomain.com 2.4.20-6 #1 thu feb 27 10:06:59
est 2003 i686 i686 i386 gnulinux '
config_args='-Uinstallusrbinperl -Dprefix=/data/gatekeeper/util
-des -Dcc=gcc'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-fno-strict-aliasing -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
optimize='-O3',
cppflags='-fno-strict-aliasing -I/usr/include/gdbm'
ccversion='', gccversion='3.2.2 20030222 (Red Hat Linux 3.2.2-5)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lgdbm -ldb -ldl -lm -lc -lcrypt -lutil
perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil
libc=/lib/libc-2.3.2.so, so=so, useshrplib=false,
libperl=libperl.a
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-rdynamic'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Nov 20 2003 15:24:36
%ENV:
PERL5LIB="/data/gatekeeper/prod/cur_ver/lib/perl:"
@INC:
/data/gatekeeper/prod/cur_ver/lib/perl
/data/gatekeeper/util/lib/perl5/5.8.0/i686-linux
/data/gatekeeper/util/lib/perl5/5.8.0
/data/gatekeeper/util/lib/perl5/site_perl/5.8.0/i686-linux
/data/gatekeeper/util/lib/perl5/site_perl/5.8.0
/data/gatekeeper/util/lib/perl5/site_perl

=======================================================================
Desktop Linux configuration returning correct results.
=======================================================================
Summary of my perl5 (revision 5.0 version 8 subversion 0)
configuration:
Platform:
osname=linux, osvers=2.4.19-4gb, archname=i686-linux
uname='linux linux-host 2.4.19-4gb #1 fri sep 13 13:14:56 utc 2002
i686 unknown '
config_args='-Uinstallusrbinperl -Dprefix=/data/gatekeeper/util
-des -Dcc=gcc'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-fno-strict-aliasing -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-O3',
cppflags='-fno-strict-aliasing'
ccversion='', gccversion='3.2', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='gcc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -ldl -lm -lc -lcrypt -lutil
perllibs=-lnsl -ldl -lm -lc -lcrypt -lutil
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.2.5'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-rdynamic'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under linux
Compiled at Jul 17 2003 20:17:04
%ENV:
PERL5LIB=":/data/gatekeeper/prod/cur_ver/lib/perl"
@INC:
/data/gatekeeper/prod/cur_ver/lib/perl
/data/gatekeeper/util/lib/perl5/5.8.0/i686-linux
/data/gatekeeper/util/lib/perl5/5.8.0
/data/gatekeeper/util/lib/perl5/site_perl/5.8.0/i686-linux
/data/gatekeeper/util/lib/perl5/site_perl/5.8.0
/data/gatekeeper/util/lib/perl5/site_perl

=======================================================================
Solaris configuration returning correct results.
=======================================================================
Summary of my perl5 (revision 5.0 version 8 subversion 0)
configuration:
Platform:
osname=solaris, osvers=2.8, archname=sun4-solaris
uname='sunos solaris 5.8 generic_108528-11 sun4u sparc
sunw,ultra-5_10 '
config_args='-Dcc=gcc -B/usr/ccs/bin/'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc -B/usr/ccs/bin/', ccflags ='-fno-strict-aliasing
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O',
cppflags='-fno-strict-aliasing'
ccversion='', gccversion='3.1', gccosandvers='solaris2.8'
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='gcc -B/usr/ccs/bin/', ldflags =' -L/usr/local/lib '
libpth=/usr/local/lib /usr/lib /usr/ccs/lib
libs=-lsocket -lnsl -lgdbm -ldl -lm -lc
perllibs=-lsocket -lnsl -ldl -lm -lc
libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: USE_LARGE_FILES
Built under solaris
Compiled at Jul 22 2002 02:55:19
%ENV:
PERL5LIB=":/oss/current/rc4498/work/projects/util/usr/local/lib/perl5/site_perl/5.8.0"
@INC:
/oss/current/rc4498/work/projects/util/usr/local/lib/perl5/site_perl/5.8.0/sun4-solaris
/oss/current/rc4498/work/projects/util/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/5.8.0/sun4-solaris
/usr/local/lib/perl5/5.8.0
/usr/local/lib/perl5/site_perl/5.8.0/sun4-solaris
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl/5.6.1
/usr/local/lib/perl5/site_perl

=======================================================================
Cygwin on Windows Configuration returning correct results
=======================================================================
Summary of my perl5 (revision 5.0 version 8 subversion 0)
configuration:
Platform:
osname=cygwin, osvers=1.3.22(0.7832), archname=cygwin-multi-64int
uname='cygwin_nt-5.0 ismene 1.3.22(0.7832) 2003-03-18 09:20 i586
unknown unknown cygwin '
config_args='-de -Dmksymlinks -Dusemultiplicity -Duse64bitint
-Doptimize=-O3 -Dman3ext=3pm'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='gcc', ccflags ='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing',
optimize='-O3',
cppflags='-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing'
ccversion='', gccversion='3.2 20020927 (prerelease)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=4
alignbytes=8, prototype=define
Linker and Libraries:
ld='ld2', ldflags =' -s -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib /lib
libs=-lgdbm -ldb -lcrypt -lutil -lgdbm_compat
perllibs=-lcrypt -lutil -lgdbm_compat
libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' -s'
cccdlflags=' ', lddlflags=' -s -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_64_BIT_INT USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
Built under cygwin
Compiled at Aug 29 2003 12:06:59
%ENV:
CYGWIN="ntsec tty"
@INC:
/usr/lib/perl5/5.8.0/cygwin-multi-64int
/usr/lib/perl5/5.8.0
/usr/lib/perl5/site_perl/5.8.0/cygwin-multi-64int
/usr/lib/perl5/site_perl/5.8.0
/usr/lib/perl5/site_perl

 

[Ralph]

OK, I finally found the answer. It makes sense, but Jeez!

It seems that, because on my RedHat 9 system the LANG environment
variable was set to en_US.UTF-8. Because of this, any byte seen by
pack as being over 127, i.e. high order bit turned on, was written as
unicode in UTF-8 format. All I had to do was to set my LANG variable
to en_US and everything started working correctly.

Hopefully these posts will help someone in the future suffering from a
similar problem.

Ralph

 

[Alan J. Flavell]

OK, I finally found the answer.

Uh-uh, the usual one, it seems. I suppose this well-known problem
(easily found if only you had known that the relevant search terms
were perl, utf-8 or utf8, and redhat) was obfuscated by the fact you
were meeting it in relation to "pack", and unaware that utf-8 was
involved.

Hopefully these posts will help someone in the future suffering from a
similar problem.

To be honest, I doubt it. Those who are already aware of the problem
would spot it immediately, but those who aren't would have no idea
where to look for the many already-available answers, as you've
already demonstrated (and in the circumstances I'm not blaming you).

But 5.8.1 has removed this behaviour, so as time goes by we should be
hearing from fewer folks who've stumbled into this morass. The
above-mentioned search produced an interesting comment:

/
Because RedHat 8 defaults to setting a UTF8 locale, incoming data is
marked as UTF8, triggering perl 5.8 to switch to UTF8 algorithms.
This behaviour of perl 5.8 is documented, and is what the Linux UTF8
people and Larry Wall wanted. So RedHat should have been aware of the
implications of their choice of making RedHat 8 default to UTF8
locales.

Because most people aren't Larry Wall or the Linux UTF8 people, 5.8.1
will change to what most people expect [...]
\

- see
http://london.pm.org/pipermail/london.pm/Week-of-Mon-20030210/017009.html

all the best