64-bit / 128-bit data element type for array?

Discussion in 'Python' started by akineko, Dec 22, 2008.

  1. akineko

    akineko Guest

    Hello everyone,

    I need to handle binary files that contain 64-bit (or 128-bit in the
    furture) unsigned int data.
    Python's array seems not supporting unsigned int type beyond 32-bit
    ('L').
    I would like to use Python array as I need to make my program work on
    both big-endian machines as well as on little-endian machines.

    What is the best way to deal with 64-bit / 128-bit data elements in
    Python (must support byteswap())?
    (must be machine-independent)

    Any idea, any hints, comments would be greatly appreciated.

    Thank you!
    Aki Niimura
    akineko, Dec 22, 2008
    #1
    1. Advertising

  2. akineko

    Guest

    akineko:
    > I need to handle binary files that contain 64-bit (or 128-bit in the
    > furture) unsigned int data.
    > Python's array seems not supporting unsigned int type beyond 32-bit
    > ('L').


    I agree that it can be useful for the built-in array module to grow
    signed/unsigned 64 bit numbers.

    Numpy supports signed/unsigned 64 bit numbers too, so that may be
    enough for you:
    http://docs.scipy.org/doc/numpy/user/basics.types.html

    Regarding 128-bit numbers you may have to support them manually, with
    pairs of uint64, with numpy. They aren't much common yet.
    (D language will have cent/ucent 128 bit integral numbers, but it's
    not a common thing).

    Bye,
    bearophile
    , Dec 22, 2008
    #2
    1. Advertising

  3. akineko

    Robert Kern Guest

    akineko wrote:
    > Hello everyone,
    >
    > I need to handle binary files that contain 64-bit (or 128-bit in the
    > furture) unsigned int data.
    > Python's array seems not supporting unsigned int type beyond 32-bit
    > ('L').
    > I would like to use Python array as I need to make my program work on
    > both big-endian machines as well as on little-endian machines.
    >
    > What is the best way to deal with 64-bit / 128-bit data elements in
    > Python (must support byteswap())?
    > (must be machine-independent)


    You might give numpy a try. We support uint64 data even on 32-bit machines
    provided that your C compiler does. I haven't seen uint128 in the wild, though.

    http://numpy.scipy.org/

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Dec 22, 2008
    #3
  4. akineko

    akineko Guest

    Hello,

    bearophile and Robert, thank you for your prompt response.
    I will try NumPy (this is a good execuse to learn and to use a new
    package).

    > I haven't seen uint128 in the wild, though.


    Of course, not many applications require uinit128 as a scalar value.
    I may need to deal with 128-bit data as it is now not uncommon to have
    128-bit data bus (or even 256-bit wide and beyond) in ASICs
    (microchip) design. Unfortunately, some designs use big-endian and
    others use little-endian ...

    Thank you and Happy Holidays!
    Aki Niimura

    On Dec 22, 11:26 am, Robert Kern <> wrote:
    > akineko wrote:
    > > Hello everyone,

    >
    > > I need to handle binary files that contain 64-bit (or 128-bit in the
    > > furture) unsigned int data.
    > > Python's array seems not supporting unsigned int type beyond 32-bit
    > > ('L').
    > > I would like to use Python array as I need to make my program work on
    > > both big-endian machines as well as on little-endian machines.

    >
    > > What is the best way to deal with 64-bit / 128-bit data elements in
    > > Python (must support byteswap())?
    > > (must be machine-independent)

    >
    > You might give numpy a try. We support uint64 data even on 32-bit machines
    > provided that your C compiler does. I haven't seen uint128 in the wild, though.
    >
    > http://numpy.scipy.org/
    >
    > --
    > Robert Kern
    >
    > "I have come to believe that the whole world is an enigma, a harmless enigma
    > that is made terrible by our own mad attempt to interpret it as though it had
    > an underlying truth."
    > -- Umberto Eco
    akineko, Dec 22, 2008
    #4
  5. akineko

    Robert Kern Guest

    akineko wrote:
    > Hello,
    >
    > bearophile and Robert, thank you for your prompt response.
    > I will try NumPy (this is a good execuse to learn and to use a new
    > package).
    >
    >> I haven't seen uint128 in the wild, though.

    >
    > Of course, not many applications require uinit128 as a scalar value.
    > I may need to deal with 128-bit data as it is now not uncommon to have
    > 128-bit data bus (or even 256-bit wide and beyond) in ASICs
    > (microchip) design. Unfortunately, some designs use big-endian and
    > others use little-endian ...


    Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
    128-bit-long chunk of data?

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Dec 22, 2008
    #5
  6. akineko

    akineko Guest

    Hello Robert,

    > Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
    > 128-bit-long chunk of data?


    That is a good question.
    A 128-bit data can be anything.
    A 128-bit data can be an instrution code (VLIW machines use such wide
    instruction). A 128-bit can be a packed ascill (16 chrs).
    A 128-bit can be a descriptor (a structure of various fields).
    It is probably safe to say that only unsigned 128-bit is required.
    (I cannot think of any situations where signed 128-bit is necessary)

    Hope this answers your question.

    Aki Niimura

    On Dec 22, 1:22 pm, Robert Kern <> wrote:
    > akineko wrote:
    > > Hello,

    >
    > > bearophile and Robert, thank you for your prompt response.
    > > I will try NumPy (this is a good execuse to learn and to use a new
    > > package).

    >
    > >> I haven't seen uint128 in the wild, though.

    >
    > > Of course, not many applications require uinit128 as a scalar value.
    > > I may need to deal with 128-bit data as it is now not uncommon to have
    > > 128-bit data bus (or even 256-bit wide and beyond) in ASICs
    > > (microchip) design. Unfortunately, some designs use big-endian and
    > > others use little-endian ...

    >
    > Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
    > 128-bit-long chunk of data?
    >
    > --
    > Robert Kern
    >
    > "I have come to believe that the whole world is an enigma, a harmless enigma
    > that is made terrible by our own mad attempt to interpret it as though it had
    > an underlying truth."
    > -- Umberto Eco
    akineko, Dec 22, 2008
    #6
  7. akineko

    Robert Kern Guest

    akineko wrote:
    > Hello Robert,
    >
    >> Is that actually a 2s-complement 128-bit unsigned integer, or is it just a
    >> 128-bit-long chunk of data?

    >
    > That is a good question.
    > A 128-bit data can be anything.
    > A 128-bit data can be an instrution code (VLIW machines use such wide
    > instruction). A 128-bit can be a packed ascill (16 chrs).
    > A 128-bit can be a descriptor (a structure of various fields).
    > It is probably safe to say that only unsigned 128-bit is required.
    > (I cannot think of any situations where signed 128-bit is necessary)


    Ah, good. numpy lets you construct your own data types from the primitives.
    Since you don't actually need uint128 arithmetic, you don't need a uint128
    primitive. You can just use dtype('V16') (meaning "void, 16 bytes long") for the
    "anything" and "instruction codes" and possible as an intermediate format.
    Byteswapping will work just fine. Use can use dtype('S16') for the ASCII. If the
    structure can be described by bytes, then the structured dtypes will work just
    fine. Unfortunately, we don't support bit-fields.

    If you need bit-field support, you may want to take a look at Construct.
    Actually, you may want to look at Construct anyways.

    http://construct.wikispaces.com/

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Dec 22, 2008
    #7
  8. akineko

    akineko Guest

    Robert wrote:
    > Ah, good. numpy lets you construct your own data types from the primitives.
    > Since you don't actually need uint128 arithmetic, you don't need a uint128
    > primitive. You can just use dtype('V16') (meaning "void, 16 bytes long") ...


    Impressive.
    I installed NumPy and it worked like a charm (I used uint64).

    The above feature is quite useful as the bus size inside a microchip
    is getting wider and wider (to get higher performance).

    Again, thank you for replying to my posting.

    Happy Holidays!
    Aki Niimura
    akineko, Dec 22, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?QmlsbCBCb3Jn?=

    Convert hashed password to 128-bit binary

    =?Utf-8?B?QmlsbCBCb3Jn?=, Apr 22, 2005, in forum: ASP .Net
    Replies:
    2
    Views:
    5,346
    =?Utf-8?B?QmlsbCBCb3Jn?=
    Apr 22, 2005
  2. ern

    128-bit hex number ---> string buffer

    ern, Sep 20, 2006, in forum: C Programming
    Replies:
    14
    Views:
    876
    Elijah Cardon
    Sep 22, 2006
  3. xyz
    Replies:
    2
    Views:
    802
  4. Eric Sosman

    Re: Integer 128 != Integer 128 ??

    Eric Sosman, Oct 12, 2010, in forum: Java
    Replies:
    6
    Views:
    816
    Screamin Lord Byron
    Oct 13, 2010
  5. chankey pathak

    Re: Integer 128 != Integer 128 ??

    chankey pathak, Oct 13, 2010, in forum: Java
    Replies:
    0
    Views:
    828
    chankey pathak
    Oct 13, 2010
Loading...

Share This Page