string arrays

Discussion in 'Perl' started by B McInnes, Oct 15, 2003.

  1. B McInnes

    B McInnes Guest

    Hello, I am trying to store approximately 50- 100 million variable
    length strings into an array and of course this is not working due to
    memory constraints. I have converted the strings to unique integers
    and then stored the integers in a vec which lets me store about 60
    million before I run out of memory. My question is: has anyone else
    run into this problem and how did you solve it?

    Thanks!
     
    B McInnes, Oct 15, 2003
    #1
    1. Advertising

  2. B McInnes <> wrote:
    > Hello, I am trying to store approximately 50- 100 million variable
    > length strings into an array and of course this is not working due to
    > memory constraints.


    I don't know why you need this big array, so I can't give you a
    very thorough answer, but usually the solution involves either
    PDL or DB_File (both on CPAN).

    --
    Steve
     
    Steve Grazzini, Oct 15, 2003
    #2
    1. Advertising

  3. B McInnes

    B McInnes Guest

    Steve Grazzini <> wrote in message news:<z54jb.13667$>...
    > B McInnes <> wrote:
    > > Hello, I am trying to store approximately 50- 100 million variable
    > > length strings into an array and of course this is not working due to
    > > memory constraints.

    >
    > I don't know why you need this big array, so I can't give you a
    > very thorough answer, but usually the solution involves either
    > PDL or DB_File (both on CPAN).


    I work with large corpora and am (trying) to implement a suffix array
    in Perl. I have tried DB_File but the I/O time became a significant
    problem. I have heard of PDL but I was under the impression that was
    more for matrix operations. I will look into though. Thanks for you
    quick response!
     
    B McInnes, Oct 15, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alexandra Stehman
    Replies:
    5
    Views:
    30,683
    Chris Smith
    Jun 17, 2004
  2. Bill Reyn
    Replies:
    3
    Views:
    2,268
    Bob Hairgrove
    Jun 22, 2004
  3. Mantorok Redgormor

    initializing arrays of arrays

    Mantorok Redgormor, Sep 10, 2003, in forum: C Programming
    Replies:
    4
    Views:
    571
  4. kelvSYC

    Arrays and Pointers to Arrays

    kelvSYC, Sep 26, 2003, in forum: C Programming
    Replies:
    2
    Views:
    387
  5. Philipp
    Replies:
    21
    Views:
    1,140
    Philipp
    Jan 20, 2009
Loading...

Share This Page