efficient access to a large int32 array

[Martin DeMello]

I have a file that is a dump of a large array of 32-bit unsigned ints
(bitvectors actually). All I need is to load this into memory and have
fast, efficient access to the array members, and a few bitmasking
operations - the array will never be written to, and ideally should
not need to be bounds-checked. Should I be looking at something like
mmap or narray to do this, or would I be better served by a custom C
extension?

martin

 

[Robert Klemme]

I have a file that is a dump of a large array of 32-bit unsigned ints
(bitvectors actually). All I need is to load this into memory and have
fast, efficient access to the array members, and a few bitmasking
operations - the array will never be written to, and ideally should
not need to be bounds-checked. Should I be looking at something like
mmap or narray to do this, or would I be better served by a custom C
extension?

How large is "large"?

robert

 

[Vincent Fourmond]

I have a file that is a dump of a large array of 32-bit unsigned ints
(bitvectors actually). All I need is to load this into memory and have
fast, efficient access to the array members, and a few bitmasking
operations - the array will never be written to, and ideally should
not need to be bounds-checked. Should I be looking at something like
mmap or narray to do this, or would I be better served by a custom C
extension?

I would write my own extension and use mmap (if you're under linux) to
save trouble and memory. That's just a question of a few hours, and
you'll get high performance.

Cheers,

Vince, in a posting mood today...

 

[Martin DeMello]

How large is "large"?

Order of 1 MB - not gigantic, but not trivial either.

martin

 

[Robert Klemme]

Order of 1 MB - not gigantic, but not trivial either.

Um, I would not say 1MB is large. What accesses do you do? Do you just
access based on index? Do you often need to load this data? etc.

In your case I'd start out writing a Ruby version (either using a single
String or an Array with integers) and change that into an extension if
performance is insufficient. My 0.02 EUR...

Kind regards

robert

 

[Martin DeMello]

Um, I would not say 1MB is large. What accesses do you do? Do you just
access based on index? Do you often need to load this data? etc.

Pure array indexing, just a lot of it.

In your case I'd start out writing a Ruby version (either using a single
String or an Array with integers) and change that into an extension if
performance is insufficient. My 0.02 EUR...

Good point - it'd be the quickest thing to get up and running, if
nothing else

martin

 

[Tim Pease]

I think Ara has been hacking some package to use Narray on mmap'ed
files, which works fast and is persistent. Search the archives.

http://codeforpeople.com/lib/ruby/nmap/

You'll need to install Guy's mmap package ...

http://moulon.inra.fr/ruby/mmap.html

Blessings,
TwP