List limits

Discussion in 'Python' started by bob_smith_17280@hotmail.com, Dec 20, 2004.

  1. Guest

    How many items can be stored in a Python list? I have close to 70,000
    items... is this within a lists limits?
     
    , Dec 20, 2004
    #1
    1. Advertising

  2. It is possible to store 70,000 items in a list (try "l =
    range(70000)"), but the best way to check if you can store all the
    items you need to store is to try it. After all if they are all very
    large you might potentially run out of memory.

    Chris
     
    Christopher De Vries, Dec 20, 2004
    #2
    1. Advertising

  3. Jeff Epler Guest

    I'm referring to Python 2.2's C headers as I answer this question. I
    believe some of may have changed by 2.4.

    The number of elements in a "variable-sized object" (those with
    Py_VAR_HEAD; I believe this includes lists, tuples, and strings) is
    stored in a platform 'int'.

    On most (desktop) systems, this means the limit of any sized object is
    no more than 2**31-1, or about 2 billion.

    On those same systems, a list object of length 128 million, give or
    take, approximately 512 megabytes of memory will be allocated for the
    list object itself (4 bytes for each pointer-to-element). If each
    element of the list is distinct, those objects will each require
    additional memory---The smallest useful Python object takes around 16
    bytes, IIRC, which would bring the total memory required to around 2560
    megabytes if I didn't screw up my arithmetic. This is close to (or
    over, depending on the system) the maximum amount of physical RAM these
    machines can accomodate, and the maximum amount of address space
    available to a single program.

    While performing list-resizing operations, there may be a temporary need
    for two copies of the list object itself, bumping the memory used up to 3
    gigs.

    Finally, the speed of some operations (l.index(item), l.pop(0),
    l.insert(0, item)) are related linearly to the size of the list, so your
    program may slow down as the lists it manipulates grow. Others, such as
    l, l.pop(), and l.append(item), are constant-time or amortized-constant-
    time.

    Jeff

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.2.1 (GNU/Linux)

    iD8DBQFBxw6pJd01MZaTXX0RAljUAJoD8/bR3hBgQOb3I40h+yKm5r9PqQCdEiBo
    s1rX4NrllE9a+vWoQQpqFe4=
    =F8Db
    -----END PGP SIGNATURE-----
     
    Jeff Epler, Dec 20, 2004
    #3
  4. > How many items can be stored in a Python list? I have close to 70,000
    > items... is this within a lists limits?


    Lists store pointers to objects. Unless you have a lot of duplicates, it is the
    objects themselves that will consume most of your memory. The list itself will
    likely be small in comparison.


    Raymond Hettinger
     
    Raymond Hettinger, Dec 20, 2004
    #4
  5. Nick Coghlan Guest

    Raymond Hettinger wrote:
    >>How many items can be stored in a Python list? I have close to 70,000
    >>items... is this within a lists limits?

    >
    >
    > Lists store pointers to objects. Unless you have a lot of duplicates, it is the
    > objects themselves that will consume most of your memory. The list itself will
    > likely be small in comparison.


    Given the size of the counter, is it actually physically possible for a list to
    run out of room before the application runs out memory?

    Even list(None for x in xrange(sys.maxint)) wouldn't do the trick, since each of
    those pointers to None is taking 4 bytes of memory, and Python's internal
    structures are already chewing up some of the address space.

    Cheers,
    Nick.

    --
    Nick Coghlan | | Brisbane, Australia
    ---------------------------------------------------------------
    http://boredomandlaziness.skystorm.net
     
    Nick Coghlan, Dec 21, 2004
    #5
  6. Nick Coghlan wrote:

    > Given the size of the counter, is it actually physically possible for a list to run out of room
    > before the application runs out memory?


    depends on the system architecture, of course: consider an 64-bit computer
    with 32-bit integers and 256 GB of memory...

    </F>
     
    Fredrik Lundh, Dec 21, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. jack
    Replies:
    0
    Views:
    6,117
  2. Ashish
    Replies:
    1
    Views:
    3,155
    Natty Gur
    Nov 19, 2003
  3. Jerry Camel

    File Upload Limits

    Jerry Camel, Feb 20, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    381
    Ken Cox [Microsoft MVP]
    Feb 21, 2004
  4. Joe Van Meer
    Replies:
    2
    Views:
    643
    Peter O'Reilly
    May 5, 2004
  5. =?Utf-8?B?cGJ1bQ==?=

    GDI+ - limits for pixel dimensions in a bitmap object

    =?Utf-8?B?cGJ1bQ==?=, Jun 18, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    362
    =?Utf-8?B?cGJ1bQ==?=
    Jun 18, 2004
Loading...

Share This Page