How big can an array be ?

Discussion in 'Perl Misc' started by Andrew Rich \(VK4TEC\), Nov 2, 2003.

  1. Is there any limitation to a PERL array size ?

    If I was to load a MySQL database into an array...

    or

    open (FILE,"/somemonsterfile"):
    while (<FILE>)
    {
    push @array, $_
    }
     
    Andrew Rich \(VK4TEC\), Nov 2, 2003
    #1
    1. Advertising

  2. In article <>, Andrew Rich (VK4TEC) wrote:
    > Is there any limitation to a PERL array size ?
    >
    > If I was to load a MySQL database into an array...
    >
    > or
    >
    > open (FILE,"/somemonsterfile"):
    > while (<FILE>)
    > {
    > push @array, $_
    > }



    You probably can't load more into memory than what can be fit
    into primary memory. Some operating systems may possibly let
    you load enough to fill out both primary and [disk based]
    virtual memory.


    However, most of the time you should work on large files line by
    line. There is very seldom any need to slurp a whole monster
    file into memory at once, especially not databases since the
    whole point of the database is to query it with a structured
    query and parse the result row by row.


    --
    Andreas Kähäri
     
    Andreas Kahari, Nov 2, 2003
    #2
    1. Advertising

  3. On Sun, 2 Nov 2003 17:57:01 +1000
    "Andrew Rich \(VK4TEC\)" <> wrote:

    > Is there any limitation to a PERL array size ?
    >
    > If I was to load a MySQL database into an array...


    Stop right there.

    _Why_ are you trying to load a whole MySQL database into an array?
    Why not use the DBI module to do whatever it is you want to do?

    --
    Jim

    Copyright notice: all code written by the author in this post is
    released under the GPL. http://www.gnu.org/licenses/gpl.txt
    for more information.

    a fortune quote ...
    Just remember: when you go to court, you are trusting your fate
    to twelve people that weren't smart enough to get out of jury
    duty!
     
    James Willmore, Nov 2, 2003
    #3
  4. -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    - -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    "Andrew Rich \(VK4TEC\)" <> wrote in news:3fa4b8de_1
    @news.iprimus.com.au:

    > Is there any limitation to a PERL array size ?
    >
    > If I was to load a MySQL database into an array...
    >
    > or
    >
    > open (FILE,"/somemonsterfile"):
    > while (<FILE>)
    > {
    > push @array, $_
    > }


    Well, for starters, that's a fairly inefficient way to load a file into an
    array. Try @array = <FILE>.

    The answer to your question is: the limit is how much virtual memory you
    have. Bear in mind that scalars (each array element is a scalar) have a
    certain amount of memory overhead, and arrays themselves have some
    overhead. So you can't load a 1Gb file into an array if you have 1Gb of
    virtual memory.

    The real answer, however, is almost certainly: "You're going about this the
    wrong way." Why on earth you would read a huge file into memory is beyond
    me. There are surely better ways to access your data. (Especially if it's
    a MySQL database!)

    - - --
    Eric
    $_ = reverse sort $ /. r , qw p ekca lre uJ reh
    ts p , map $ _. $ " , qw e p h tona e and print

    - -----BEGIN PGP SIGNATURE-----
    Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

    iQA/AwUBP6WRD2PeouIeTNHoEQI1aACglkr5Yu7sXEMLXGkoB5yU05kwH6wAoPBn
    mXhGm5GvfB44hj9p9QDJrchR
    =N7/k
    - -----END PGP SIGNATURE-----

    -----BEGIN PGP SIGNATURE-----
    Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

    iQA/AwUBP6WRGWPeouIeTNHoEQJ8/wCeKkS0J1PHgAQ2FMCZpGM+a5J7noQAnAwc
    Fng2S3wGhe35p1/V2o4slmQa
    =TL1u
    -----END PGP SIGNATURE-----
     
    Eric J. Roode, Nov 2, 2003
    #4
  5. Andreas Kahari <> wrote in message news:<>...
    > In article <>, Andrew Rich (VK4TEC) wrote:
    > > Is there any limitation to a PERL array size ?
    > >
    > > If I was to load a MySQL database into an array...
    > >
    > > or
    > >
    > > open (FILE,"/somemonsterfile"):
    > > while (<FILE>)
    > > {
    > > push @array, $_
    > > }

    >
    >
    > You probably can't load more into memory than what can be fit
    > into primary memory. Some operating systems may possibly let
    > you load enough to fill out both primary and [disk based]
    > virtual memory.
    >
    >
    > However, most of the time you should work on large files line by
    > line. There is very seldom any need to slurp a whole monster
    > file into memory at once, especially not databases since the
    > whole point of the database is to query it with a structured
    > query and parse the result row by row.



    Using linux I was able to load a hash, where the process was taking up
    ~ 1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
    imagine that on some OS's you might not be able to occupy more that
    4GB of RAM, since pointers would go over their limit (for 32 bit
    pointers at least). I was also root while doing this, I don't know if
    some users are prohibited more than others. Trying this non-root on
    Solaris, I was told that I ran out of memory much earlier. (I don't
    know why ulimit maybey???)

    *** I had 2 huge files to cross reference with each other.
    I originally tried doing the calculation with both
    a relational database (Informix) and with Berkely DB.
    Loading all of one file into memory and using that in memory hash
    for processing against the 2nd file was at least 10 times faster
    than using a database. ***
     
    Bryan Castillo, Nov 3, 2003
    #5
  6. Andrew Rich \(VK4TEC\)

    Keith Keller Guest

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    On 2003-11-03, Bryan Castillo <> wrote:
    > *** I had 2 huge files to cross reference with each other.
    > I originally tried doing the calculation with both
    > a relational database (Informix) and with Berkely DB.
    > Loading all of one file into memory and using that in memory hash
    > for processing against the 2nd file was at least 10 times faster
    > than using a database. ***


    Well, the OP already has MySQL, so it might not be so helpful to dump
    his query results to a file then process it.

    - --keith

    - --
    -francisco.ca.us
    (try just my userid to email me)
    AOLSFAQ=http://wombat.san-francisco.ca.us/cgi-bin/fom

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.3 (GNU/Linux)

    iD8DBQE/pggBhVcNCxZ5ID8RApydAJ44U6IfxFwxUvX3eUmQZA2zdq6wZwCcDHx4
    hX3l+19ipf+PjHkbnK8S9s8=
    =6phz
    -----END PGP SIGNATURE-----
     
    Keith Keller, Nov 3, 2003
    #6
  7. Andrew Rich \(VK4TEC\)

    Tore Aursand Guest

    On Sun, 02 Nov 2003 17:57:01 +1000, Andrew Rich (VK4TEC) wrote:
    > Is there any limitation to a PERL array size ?


    Not really. It depends on the OS and how much memory the computer has.

    > If I was to load a MySQL database into an array...


    Why would you do that?

    > open (FILE,"/somemonsterfile"):
    > while (<FILE>)
    > {
    > push @array, $_
    > }


    You might as well write

    open( FILE, '/somemonsterfile' ) or die "$!\n";
    @array = <FILE>;
    close( FILE );

    But you really need to read the whole file in one take? Are you
    absolutely sure you can't process one line at a time?


    --
    Tore Aursand <>
     
    Tore Aursand, Nov 3, 2003
    #7
  8. [This followup was posted to comp.lang.perl.misc ]

    In article <>, "Andrew Rich \(VK4TEC\)"
    <> says...
    > Is there any limitation to a PERL array size ?
    >
    > If I was to load a MySQL database into an array...
    >
    > or
    >
    > open (FILE,"/somemonsterfile"):
    > while (<FILE>)
    > {
    > push @array, $_
    > }



    Perl by definition puts no limit on the size of an array.
    However there may be limit opposed by your operating system as to the
    amount of memory that can be attached to a process.
     
    Master Web Surfer, Nov 3, 2003
    #8
  9. Andrew Rich \(VK4TEC\)

    Guest

    (Bryan Castillo) wrote:
    >
    > Using linux I was able to load a hash, where the process was taking up
    > ~ 1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
    > imagine that on some OS's you might not be able to occupy more that
    > 4GB of RAM, since pointers would go over their limit (for 32 bit
    > pointers at least). I was also root while doing this, I don't know if
    > some users are prohibited more than others. Trying this non-root on
    > Solaris, I was told that I ran out of memory much earlier. (I don't
    > know why ulimit maybey???)
    >
    > *** I had 2 huge files to cross reference with each other.
    > I originally tried doing the calculation with both
    > a relational database (Informix) and with Berkely DB.
    > Loading all of one file into memory and using that in memory hash
    > for processing against the 2nd file was at least 10 times faster
    > than using a database. ***


    Did you try a sort-merge? I've never seen a hash that doesn't fit in
    physical memory be fast about anything.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service New Rate! $9.95/Month 50GB
     
    , Nov 3, 2003
    #9
  10. Andrew Rich \(VK4TEC\)

    Guest

    Master Web Surfer <> wrote:
    > [This followup was posted to comp.lang.perl.misc ]
    >
    > In article <>, "Andrew Rich \(VK4TEC\)"
    > <> says...
    > > Is there any limitation to a PERL array size ?
    > >
    > > If I was to load a MySQL database into an array...
    > >
    > > or
    > >
    > > open (FILE,"/somemonsterfile"):
    > > while (<FILE>)
    > > {
    > > push @array, $_
    > > }

    >
    > Perl by definition puts no limit on the size of an array.


    By the definition of what?

    It seems that the index of an array must fit in a 32-bit integer, at least
    in some systems, and probably a 64-bit integer on the rest. So that is
    a limit. A limit that will probably never be reached in physical memory,
    but still a limit.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service New Rate! $9.95/Month 50GB
     
    , Nov 3, 2003
    #10
  11. Andrew Rich \(VK4TEC\)

    Eric Wilhelm Guest

    On Sun, 02 Nov 2003 23:43:40 -0600, Bryan Castillo wrote:

    > Using linux I was able to load a hash, where the process was taking up ~
    > 1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
    > imagine that on some OS's you might not be able to occupy more that 4GB
    > of RAM, since pointers would go over their limit (for 32 bit pointers at
    > least).


    It is my understanding that any one process under Linux is limited to the
    memory which is addressable by the processor. If you have a 32bit Intel
    architecture, this comes to something around 4GB. This seems to
    correspond to the "Out of Memory!" messages which I have sometimes
    received.

    I have not tried this (and the IPC would probably be tricky), but I think
    fork()ing a few more processes would allow you to expand the memory
    allowed by your program (except that each process would still be limited,
    so it would have to dump any pre-existing memory and then have a lot of
    message-passing (of course, we have all started looking for ways to
    rewrite the code by this point.))

    --Eric
     
    Eric Wilhelm, Nov 4, 2003
    #11
  12. wrote:
    > Master Web Surfer <> wrote:
    >> [This followup was posted to comp.lang.perl.misc ]
    >>
    >> In article <>, "Andrew Rich \(VK4TEC\)"
    >> <> says...
    >>> Is there any limitation to a PERL array size ?

    [...]
    >> Perl by definition puts no limit on the size of an array.

    >
    > By the definition of what?


    By definition of Perl.

    > It seems that the index of an array must fit in a 32-bit integer, at
    > least in some systems, and probably a 64-bit integer on the rest. So
    > that is a limit.


    But that is a limit that would be imposed by perl (or the OS perl is running
    on), not by Perl.
    Of course, there is no PERL, so we can't really tell if the OP was inquiring
    about limits in Perl or in perl.

    jue
     
    Jürgen Exner, Nov 4, 2003
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    345
    Larry I Smith
    Jun 21, 2005
  2. Shaguf
    Replies:
    0
    Views:
    525
    Shaguf
    Dec 24, 2008
  3. Shaguf
    Replies:
    0
    Views:
    482
    Shaguf
    Dec 26, 2008
  4. Shaguf
    Replies:
    0
    Views:
    261
    Shaguf
    Dec 26, 2008
  5. Shaguf
    Replies:
    0
    Views:
    238
    Shaguf
    Dec 24, 2008
Loading...

Share This Page