Easy way to make Numarray Bool use 1 bit?

A

andrewfelch

I need to be able to store thousands of one-million-length bit arrays.
The first operations use the & operator. But eventually I must perform
an inner product with a one-million-length integer array (summing at
all the indices of the integer operand where the boolean operand has a
"1").

If I use the Boolean type, I end up using 1-byte, per needed bit, which
is not efficient enough (only 1,000 arrays per gigabyte of memory).
I'm limited by 32-bit windows to use only 2 GB total.

Anybody know of a fast way to use only 125 kilobytes of memory per
array and yet be able to inner product with an integer array quickly?
Don't tell me I have to write C/C++ ! (ugh!)

TIA,
AndrewF
 
A

andrewfelch

Ahh, I figured it out. The key is to set up the shapes so they
broadcast like an outerproduct, but using bitwise_and as the operation
and the second operand as the bit filter. So we temporarily expand
each bit to a byte for numarray to be able to use it for dot products.
If you use a temporary Boolean variable to store the bitwise_and ufunc
output, the answer is automatically converted to boolean.

Set up the shapes for an innerproduct-like operation to go from byte
format back to bit compact Uint8 format. Go numarray!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top