64-bit / 128-bit data element type for array?

A

akineko

Hello everyone,

I need to handle binary files that contain 64-bit (or 128-bit in the
furture) unsigned int data.
Python's array seems not supporting unsigned int type beyond 32-bit
('L').
I would like to use Python array as I need to make my program work on
both big-endian machines as well as on little-endian machines.

What is the best way to deal with 64-bit / 128-bit data elements in
Python (must support byteswap())?
(must be machine-independent)

Any idea, any hints, comments would be greatly appreciated.

Thank you!
Aki Niimura
 
B

bearophileHUGS

akineko:
I need to handle binary files that contain 64-bit (or 128-bit in the
furture) unsigned int data.
Python's array seems not supporting unsigned int type beyond 32-bit
('L').

I agree that it can be useful for the built-in array module to grow
signed/unsigned 64 bit numbers.

Numpy supports signed/unsigned 64 bit numbers too, so that may be
enough for you:
http://docs.scipy.org/doc/numpy/user/basics.types.html

Regarding 128-bit numbers you may have to support them manually, with
pairs of uint64, with numpy. They aren't much common yet.
(D language will have cent/ucent 128 bit integral numbers, but it's
not a common thing).

Bye,
bearophile
 
R

Robert Kern

akineko said:
Hello everyone,

I need to handle binary files that contain 64-bit (or 128-bit in the
furture) unsigned int data.
Python's array seems not supporting unsigned int type beyond 32-bit
('L').
I would like to use Python array as I need to make my program work on
both big-endian machines as well as on little-endian machines.

What is the best way to deal with 64-bit / 128-bit data elements in
Python (must support byteswap())?
(must be machine-independent)

You might give numpy a try. We support uint64 data even on 32-bit machines
provided that your C compiler does. I haven't seen uint128 in the wild, though.

http://numpy.scipy.org/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
A

akineko

Hello,

bearophile and Robert, thank you for your prompt response.
I will try NumPy (this is a good execuse to learn and to use a new
package).
I haven't seen uint128 in the wild, though.

Of course, not many applications require uinit128 as a scalar value.
I may need to deal with 128-bit data as it is now not uncommon to have
128-bit data bus (or even 256-bit wide and beyond) in ASICs
(microchip) design. Unfortunately, some designs use big-endian and
others use little-endian ...

Thank you and Happy Holidays!
Aki Niimura
 
R

Robert Kern

akineko said:
Hello,

bearophile and Robert, thank you for your prompt response.
I will try NumPy (this is a good execuse to learn and to use a new
package).


Of course, not many applications require uinit128 as a scalar value.
I may need to deal with 128-bit data as it is now not uncommon to have
128-bit data bus (or even 256-bit wide and beyond) in ASICs
(microchip) design. Unfortunately, some designs use big-endian and
others use little-endian ...

Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
128-bit-long chunk of data?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
A

akineko

Hello Robert,
Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
128-bit-long chunk of data?

That is a good question.
A 128-bit data can be anything.
A 128-bit data can be an instrution code (VLIW machines use such wide
instruction). A 128-bit can be a packed ascill (16 chrs).
A 128-bit can be a descriptor (a structure of various fields).
It is probably safe to say that only unsigned 128-bit is required.
(I cannot think of any situations where signed 128-bit is necessary)

Hope this answers your question.

Aki Niimura
 
R

Robert Kern

akineko said:
Hello Robert,


That is a good question.
A 128-bit data can be anything.
A 128-bit data can be an instrution code (VLIW machines use such wide
instruction). A 128-bit can be a packed ascill (16 chrs).
A 128-bit can be a descriptor (a structure of various fields).
It is probably safe to say that only unsigned 128-bit is required.
(I cannot think of any situations where signed 128-bit is necessary)

Ah, good. numpy lets you construct your own data types from the primitives.
Since you don't actually need uint128 arithmetic, you don't need a uint128
primitive. You can just use dtype('V16') (meaning "void, 16 bytes long") for the
"anything" and "instruction codes" and possible as an intermediate format.
Byteswapping will work just fine. Use can use dtype('S16') for the ASCII. If the
structure can be described by bytes, then the structured dtypes will work just
fine. Unfortunately, we don't support bit-fields.

If you need bit-field support, you may want to take a look at Construct.
Actually, you may want to look at Construct anyways.

http://construct.wikispaces.com/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
A

akineko

Robert said:
Ah, good. numpy lets you construct your own data types from the primitives.
Since you don't actually need uint128 arithmetic, you don't need a uint128
primitive. You can just use dtype('V16') (meaning "void, 16 bytes long") ...

Impressive.
I installed NumPy and it worked like a charm (I used uint64).

The above feature is quite useful as the bus size inside a microchip
is getting wider and wider (to get higher performance).

Again, thank you for replying to my posting.

Happy Holidays!
Aki Niimura
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top