Newbie question about memory management

[Oh Kyu Yoon]

I am writing my first python program, and I have to deal with several lists
of double precision numbers that are about 10,000,000 in length.
If I have 5 such lists, 5 x 10,000,000 x 8bytes ~ 400 MB which is too much
for the computer memory.
Does anyone have better strategies to do this?
Thanks.

 

[Irmen de Jong]

I am writing my first python program, and I have to deal with several lists
of double precision numbers that are about 10,000,000 in length.
If I have 5 such lists, 5 x 10,000,000 x 8bytes ~ 400 MB which is too much
for the computer memory.
Does anyone have better strategies to do this?

Yes; keep them on disk. All of them, most of them, or partially,
you don't tell us what you're doing with the lists, and so it
is hard to guess what would be the best approach.

--Irmen.

P.S. also look at the 'array' module.

 

[Terry Reedy]

I am writing my first python program, and I have to deal with several lists
of double precision numbers that are about 10,000,000 in length.
If I have 5 such lists, 5 x 10,000,000 x 8bytes ~ 400 MB which is too much
for the computer memory.
Does anyone have better strategies to do this?

If you use Python lists of wrapped floats, you will use more than 8 bytes
each. To make arrays of C floats, and for much faster computation, use
Numerical Python. It includes wrappings of LinPak and FFTPak.

tjr

 

[Emile van Sebille]

Oh Kyu Yoon:

I am writing my first python program, and I have to deal with several lists
of double precision numbers that are about 10,000,000 in length.
If I have 5 such lists, 5 x 10,000,000 x 8bytes ~ 400 MB which is too much
for the computer memory.
Does anyone have better strategies to do this?

How about numeric with Kragen Sitaker's memory mapped files.
http://www.pfdubois.com/numpy/
http://pobox.com/~kragen/sw/arrayfrombuffer/

HTH,

Emile van Sebille
(e-mail address removed)