Perl Regex - Hex bytes

[JEB]

I am trying to use Perl to rescue some legacy word processor files.
The files are ascii, except that some control codes use
bytes in the $80-$ff ranges. I slurp the file into a string for editing.

Regex can hand the bytes <\x7f, but fails to recognize bytes that are \x80
or above.

e.g.,

/\x03//; works
/\x81//; doesn't

Since I thought the problem might be related the adoption of unicode, I've
tried various things like;

no encoding;
use bytes;
and various forms of encoding;
etc.

Nothing helped, but I may not have done it right.

I'm using Perl 5.8+(whatever the lastest revision is) with Redhat Linux
8.0.

Is this something a Perl regex just can't handle?

JEB

 

[Rafael Garcia-Suarez]

/\x03//; works
/\x81//; doesn't

You're giving too little information.
Could you post a sample code that demonstrates the problem, along with
your definition of "doesn't work" ? (warnings, error, expected result vs
actual result)

 

[Alan J. Flavell]

I'm using Perl 5.8+(whatever the lastest revision is)

I suspect you're really using 5.8.0 (as opposed to 5.8.1).
with Redhat Linux

8.0.

I think that's your clue. Look for utf-8 in your linux locale
setting. It's confusing Perl 5.8.0 into using unicode mode.

(And read other discussions and FAQs on this issue).

Either change your locale setting to remove the reference
to utf-8 (I'm sure this works); or upgrade to 5.8.1, where this
coupling between locale and Perl default behaviour was found too
confusing and has been removed (so I'm told).

Is this something a Perl regex just can't handle?

Wrong diagnosis. Certainly it can handle it.

 

[JEB]

I think that's your clue. Look for utf-8 in your linux locale
setting. It's confusing Perl 5.8.0 into using unicode mode.

(And read other discussions and FAQs on this issue).

Either change your locale setting to remove the reference
to utf-8 (I'm sure this works); or upgrade to 5.8.1, where this
coupling between locale and Perl default behaviour was found too
confusing and has been removed (so I'm told).

THANKS for the idea and help.

Exporting LC_ALL="en_US" in /etc/profile fixed the problem, though in a
clumsy way. I hope it doesn't create problems elsewhere.

JEB

 

[Ben Morrow]

Exporting LC_ALL="en_US" in /etc/profile fixed the problem, though in a
clumsy way. I hope it doesn't create problems elsewhere.

Installing 5.8.1 will also fix it, without the need to lose your
Unicode locale. Alternatively, as a temporary fix, you could

1. Make sure you have /usr/bin/perl5.8.0: if not, copy it from
/usr/bin/perl
2. Remove /usr/bin/perl
3. Create a shell script /usr/bin/perl containing
#!/bin/sh
export LC_ALL="en_US.ISO8859-1"
exec /usr/bin/perl5.8.0 "$@"

Yes, I think this is a pretty evil hack, too, but if you have problems
with losing the Unicode locale it may help.

Ben