string arrays

[B McInnes]

Hello, I am trying to store approximately 50- 100 million variable
length strings into an array and of course this is not working due to
memory constraints. I have converted the strings to unique integers
and then stored the integers in a vec which lets me store about 60
million before I run out of memory. My question is: has anyone else
run into this problem and how did you solve it?

Thanks!

 

[Steve Grazzini]

Hello, I am trying to store approximately 50- 100 million variable
length strings into an array and of course this is not working due to
memory constraints.

I don't know why you need this big array, so I can't give you a
very thorough answer, but usually the solution involves either
PDL or DB_File (both on CPAN).

 

[B McInnes]

I don't know why you need this big array, so I can't give you a
very thorough answer, but usually the solution involves either
PDL or DB_File (both on CPAN).

I work with large corpora and am (trying) to implement a suffix array
in Perl. I have tried DB_File but the I/O time became a significant
problem. I have heard of PDL but I was under the impression that was
more for matrix operations. I will look into though. Thanks for you
quick response!