How big can an array be ?

[Andrew Rich \(VK4TEC\)]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

 

[Andreas Kahari]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

You probably can't load more into memory than what can be fit
into primary memory. Some operating systems may possibly let
you load enough to fill out both primary and [disk based]
virtual memory.


However, most of the time you should work on large files line by
line. There is very seldom any need to slurp a whole monster
file into memory at once, especially not databases since the
whole point of the database is to query it with a structured
query and parse the result row by row.

 

[James Willmore]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

Stop right there.

_Why_ are you trying to load a whole MySQL database into an array?
Why not use the DBI module to do whatever it is you want to do?

--
Jim

Copyright notice: all code written by the author in this post is
released under the GPL. http://www.gnu.org/licenses/gpl.txt
for more information.

a fortune quote ...
Just remember: when you go to court, you are trusting your fate
to twelve people that weren't smart enough to get out of jury
duty!

 

[Eric J. Roode]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

Well, for starters, that's a fairly inefficient way to load a file into an
array. Try @array = <FILE>.

The answer to your question is: the limit is how much virtual memory you
have. Bear in mind that scalars (each array element is a scalar) have a
certain amount of memory overhead, and arrays themselves have some
overhead. So you can't load a 1Gb file into an array if you have 1Gb of
virtual memory.

The real answer, however, is almost certainly: "You're going about this the
wrong way." Why on earth you would read a huge file into memory is beyond
me. There are surely better ways to access your data. (Especially if it's
a MySQL database!)

- - --
Eric
$_ = reverse sort $ /. r , qw p ekca lre uJ reh
ts p , map $ _. $ " , qw e p h tona e and print

- -----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBP6WRD2PeouIeTNHoEQI1aACglkr5Yu7sXEMLXGkoB5yU05kwH6wAoPBn
mXhGm5GvfB44hj9p9QDJrchR
=N7/k
- -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBP6WRGWPeouIeTNHoEQJ8/wCeKkS0J1PHgAQ2FMCZpGM+a5J7noQAnAwc
Fng2S3wGhe35p1/V2o4slmQa
=TL1u
-----END PGP SIGNATURE-----

 

[Bryan Castillo]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

You probably can't load more into memory than what can be fit
into primary memory. Some operating systems may possibly let
you load enough to fill out both primary and [disk based]
virtual memory.


However, most of the time you should work on large files line by
line. There is very seldom any need to slurp a whole monster
file into memory at once, especially not databases since the
whole point of the database is to query it with a structured
query and parse the result row by row.

Using linux I was able to load a hash, where the process was taking up
~ 1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
imagine that on some OS's you might not be able to occupy more that
4GB of RAM, since pointers would go over their limit (for 32 bit
pointers at least). I was also root while doing this, I don't know if
some users are prohibited more than others. Trying this non-root on
Solaris, I was told that I ran out of memory much earlier. (I don't
know why ulimit maybey???)

*** I had 2 huge files to cross reference with each other.
I originally tried doing the calculation with both
a relational database (Informix) and with Berkely DB.
Loading all of one file into memory and using that in memory hash
for processing against the 2nd file was at least 10 times faster
than using a database. ***

 

[Keith Keller]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

*** I had 2 huge files to cross reference with each other.
I originally tried doing the calculation with both
a relational database (Informix) and with Berkely DB.
Loading all of one file into memory and using that in memory hash
for processing against the 2nd file was at least 10 times faster
than using a database. ***

Well, the OP already has MySQL, so it might not be so helpful to dump
his query results to a file then process it.

- --keith

- --
(e-mail address removed)-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://wombat.san-francisco.ca.us/cgi-bin/fom

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/pggBhVcNCxZ5ID8RApydAJ44U6IfxFwxUvX3eUmQZA2zdq6wZwCcDHx4
hX3l+19ipf+PjHkbnK8S9s8=
=6phz
-----END PGP SIGNATURE-----

 

[Tore Aursand]

Is there any limitation to a PERL array size ?

Not really. It depends on the OS and how much memory the computer has.

If I was to load a MySQL database into an array...

Why would you do that?

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

You might as well write

open( FILE, '/somemonsterfile' ) or die "$!\n";
@array = <FILE>;
close( FILE );

But you really need to read the whole file in one take? Are you
absolutely sure you can't process one line at a time?

 

[Master Web Surfer]

[This followup was posted to comp.lang.perl.misc ]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

Perl by definition puts no limit on the size of an array.
However there may be limit opposed by your operating system as to the
amount of memory that can be attached to a process.

 

[ctcgag]

Using linux I was able to load a hash, where the process was taking up
~ 1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
imagine that on some OS's you might not be able to occupy more that
4GB of RAM, since pointers would go over their limit (for 32 bit
pointers at least). I was also root while doing this, I don't know if
some users are prohibited more than others. Trying this non-root on
Solaris, I was told that I ran out of memory much earlier. (I don't
know why ulimit maybey???)

*** I had 2 huge files to cross reference with each other.
I originally tried doing the calculation with both
a relational database (Informix) and with Berkely DB.
Loading all of one file into memory and using that in memory hash
for processing against the 2nd file was at least 10 times faster
than using a database. ***

Did you try a sort-merge? I've never seen a hash that doesn't fit in
physical memory be fast about anything.

Xho

 

[ctcgag]

[This followup was posted to comp.lang.perl.misc ]

Is there any limitation to a PERL array size ?

If I was to load a MySQL database into an array...

or

open (FILE,"/somemonsterfile"):
while (<FILE>)
{
push @array, $_
}

Perl by definition puts no limit on the size of an array.

By the definition of what?

It seems that the index of an array must fit in a 32-bit integer, at least
in some systems, and probably a 64-bit integer on the rest. So that is
a limit. A limit that will probably never be reached in physical memory,
but still a limit.

Xho

 

[Eric Wilhelm]

Using linux I was able to load a hash, where the process was taking up ~
1GB of memory. I only had 512 MB of RAM. I had 1 GB of swap. I
imagine that on some OS's you might not be able to occupy more that 4GB
of RAM, since pointers would go over their limit (for 32 bit pointers at
least).

It is my understanding that any one process under Linux is limited to the
memory which is addressable by the processor. If you have a 32bit Intel
architecture, this comes to something around 4GB. This seems to
correspond to the "Out of Memory!" messages which I have sometimes
received.

I have not tried this (and the IPC would probably be tricky), but I think
fork()ing a few more processes would allow you to expand the memory
allowed by your program (except that each process would still be limited,
so it would have to dump any pre-existing memory and then have a lot of
message-passing (of course, we have all started looking for ways to
rewrite the code by this point.))

--Eric

 

[Jürgen Exner]

[This followup was posted to comp.lang.perl.misc ]

Is there any limitation to a PERL array size ?

[...]
Perl by definition puts no limit on the size of an array.

By the definition of what?

By definition of Perl.

It seems that the index of an array must fit in a 32-bit integer, at
least in some systems, and probably a 64-bit integer on the rest. So
that is a limit.

But that is a limit that would be imposed by perl (or the OS perl is running
on), not by Perl.
Of course, there is no PERL, so we can't really tell if the OP was inquiring
about limits in Perl or in perl.

jue