Unintuitive expression evaluation

[Bill Davidsen]

I am reading a log file which has 32 bit unsigned values, and need to
take the difference between them. If the difference is negative I assume
that the data counter has rolled over, and I want to add 231 to get the
correct value. For readability I wanted to put the value in a variable
so it would be obvious what was happening.

So I wrote:
$Roll32 = (1 << 32); # the way I would for a C macro
but the value was (after I did some looking) one! Then I wrote:
$Roll32 = 1 << 32; # in case the parens were a issue
bit it was still one, so I wrote:
$Roll32 = ( 4 * (1 << 30) ); # which works (4G)

This is the first time perl has failed to do int=>double conversion when
expected (by me). Just a note in case someone else is ever doing
similar, perl on Linux 32 bit, v5.6.1, v5.8.0, v5.8.5 builds.

 

[A. Sinan Unur]

I am reading a log file which has 32 bit unsigned values, and need to
take the difference between them. If the difference is negative I
assume that the data counter has rolled over, and I want to add 231 to
get the correct value. For readability I wanted to put the value in a
variable so it would be obvious what was happening.

So I wrote:
$Roll32 = (1 << 32); # the way I would for a C macro
but the value was (after I did some looking) one! Then I wrote:
$Roll32 = 1 << 32; # in case the parens were a issue
bit it was still one, so I wrote:
$Roll32 = ( 4 * (1 << 30) ); # which works (4G)

This is the first time perl has failed to do int=>double conversion
when expected (by me).

The only thing that failed in this case is you.

Just a note in case someone else is ever doing
similar, perl on Linux 32 bit, v5.6.1, v5.8.0, v5.8.5 builds.

You mean "just in case someone else refuses to read the documentation"?

perldoc perlop:

Shift Operators
....
Either way, the implementation isn't going to generate results larger
than the size of the integer type Perl was built with (32 bits or 64
bits).

The result of overflowing the range of the integers is undefined
because it is undefined also in C. In other words, using 32-bit
integers, "1 << 32" is undefined. Shifting by a negative number
of bits is also undefined.

How hard is it to read this, and understand it?

Sinan

 

[Tassilo v. Parseval]

Also sprach A. Sinan Unur:

The only thing that failed in this case is you.


You mean "just in case someone else refuses to read the documentation"?

perldoc perlop:

Shift Operators
...
Either way, the implementation isn't going to generate results larger
than the size of the integer type Perl was built with (32 bits or 64
bits).

The result of overflowing the range of the integers is undefined
because it is undefined also in C. In other words, using 32-bit
integers, "1 << 32" is undefined. Shifting by a negative number
of bits is also undefined.

How hard is it to read this, and understand it?

Considering that

$Roll32 = ( 4 * (1 << 30) );

gives the expected result, these paragraphs may be a tad difficult to
understand, particularly for people with a C-background.

Replacing a two-bit shift with a multiplication by four works because
perl internally upgrades (as needed) the scalar from holding a plain
integer to holding a double when doing the multiplication.

I think the left-shift operator could be made smarter so that it always
works up to 2**64 - 1, even on 32bit machines.

Tassilo

 

[Anno Siegel]

Also sprach A. Sinan Unur:

I think the left-shift operator could be made smarter so that it always
works up to 2**64 - 1, even on 32bit machines.

....but only if the bit pattern can be preserved. It wouldn't do to return
the numeric equivalent as a float. So there must be support for 64 bit
integers, either in hardware or in software. I don't think Perl supplies
that if the native C doesn't.

Anno

 

[Dr.Ruud]

Bill Davidsen schreef:

(1 << 32)

That is a 33-bit value.

 

[Tassilo v. Parseval]

Also sprach Anno Siegel:

...but only if the bit pattern can be preserved. It wouldn't do to return
the numeric equivalent as a float. So there must be support for 64 bit
integers, either in hardware or in software. I don't think Perl supplies
that if the native C doesn't.

Yes, the assumption was that some sort of 64bit integer exists. If it
doesn't, left-shift will remain as it is now. The patch that should do
that is:

--- perl-p-5.8.0@25859~/pp.c 2005-10-27 08:58:48.000000000 +0200
+++ perl-p-5.8.0@25859/pp.c 2005-10-28 08:42:45.000000000 +0200
@@ -1627,12 +1627,32 @@ PP(pp_left_shift)
{
const IV shift = POPi;
if (PL_op->op_private & HINT_INTEGER) {
+#if !defined(USE_64_BIT_INT) && defined(I64TYPE)
+ I64TYPE i = TOPi;
+ i <<= shift;
+ if (i > I32_MAX) {
+ SETn((NV)i);
+ SvNOK_only(TARG);
+ } else
+ SETi(i);
+#else
IV i = TOPi;
SETi(i << shift);
+#endif
}
else {
+#if !defined(USE_64_BIT_INT) && defined(U64TYPE)
+ U64TYPE u = TOPu;
+ u <<= shift;
+ if (u > U32_MAX) {
+ SETn((NV)u);
+ SvNOK_only(TARG);
+ } else
+ SETu(u);
+#else
UV u = TOPu;
SETu(u << shift);
+#endif
}
RETURN;
}

That fails two core-tests, though, and I suspect it's even
backwards-incompatible because there were oudoubtedtly blockheads out
there that assumed some sort of defined behaviour from overflowing
left-shifts in their scripts.

Nonetheless I passed that draft to the porters to see what they think
about it.

Tassilo

 

[Bill Davidsen]

The only thing that failed in this case is you.

Sorry, I didn't realize the word "unintuitive" would not be understood
by some people.

This works on 64 bit machines. This works on 32 bit machines if you
compile for 64 bit integers. This doesn't work on 32 bit machines
without 64 bit integers. However, it does work if you do it at runtime
instead of compile time:

oddball:davidsen> perl -e '
@x=(4,1);
for $a (@x) {
$b=$a<<29;
$c=$a<<30;
print "$a $b $c\n"
}'
4 2147483648 0
1 536870912 1073741824


A feature which works in some places and not in others, and which
doesn't generate compiler warnings in cases where it clearly isn't going
to work certainly qualifies as "unintuitive" for me.