checking available system memory?

Discussion in 'Python' started by Darren Dale, Oct 15, 2004.

  1. Darren Dale

    Darren Dale Guest

    I am doing linear algebra with large numarray. It is very efficient, but I
    have a small problem due to the size of my data. The dot product of a
    10,000x3 double array with a 3x6,250,000 double array will consume 500GB of
    memory. I need to break the operations up into managable chunks, so I dont
    consume all the available memory and get a segmentation fault.

    Its not a problem with numpy, I just need to intelligently slice up one of
    my arrays so my routine works within the available system resources. Are
    there any utilities that can query how much memory is available?

    Thanks,
    Darren
    Darren Dale, Oct 15, 2004
    #1
    1. Advertising

  2. Darren Dale

    Ville Vainio Guest

    >>>>> "Darren" == Darren Dale <> writes:

    Darren> Its not a problem with numpy, I just need to intelligently
    Darren> slice up one of my arrays so my routine works within the
    Darren> available system resources. Are there any utilities that
    Darren> can query how much memory is available?

    What platform? On Linux at least you can do 'cat /proc/meminfo' to get
    more info than you probably want to have...

    --
    Ville Vainio http://tinyurl.com/2prnb
    Ville Vainio, Oct 15, 2004
    #2
    1. Advertising

  3. Darren Dale <> writes:

    > I am doing linear algebra with large numarray. It is very efficient, but I
    > have a small problem due to the size of my data. The dot product of a
    > 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of
    > memory. I need to break the operations up into managable chunks, so I dont
    > consume all the available memory and get a segmentation fault.
    >
    > Its not a problem with numpy, I just need to intelligently slice up one of
    > my arrays so my routine works within the available system resources. Are
    > there any utilities that can query how much memory is available?


    Not really, it tends to quite operating-system specific.

    Instead of saying "What's the largest chunk I can do at a time", how
    about "What's the smallest chunk, where bigger chunks won't get me
    much?". If you operate on chunks that are on the order of the cache
    size of the processor, that's probably sufficient.

    Also, if you're using numarray.dot, note that it doesn't use BLAS (yet), so
    it's not as efficient as it could be if it used it (through ATLAS, for
    instance).

    --
    |>|\/|<
    /--------------------------------------------------------------------------\
    |David M. Cooke
    |cookedm(at)physics(dot)mcmaster(dot)ca
    David M. Cooke, Oct 15, 2004
    #3

  4. > I am doing linear algebra with large numarray. It is very efficient, but I
    > have a small problem due to the size of my data. The dot product of a
    > 10,000x3 double array with a 3x6,250,000 double array will consume 500GB of
    > memory. I need to break the operations up into managable chunks, so I dont
    > consume all the available memory and get a segmentation fault.
    >
    > Its not a problem with numpy, I just need to intelligently slice up one of
    > my arrays so my routine works within the available system resources. Are
    > there any utilities that can query how much memory is available?


    Unless you are running bigmem patches on linux, or the equivalent in
    windows, you are limited to 2 gigs of memory per process.

    How much memory do you really have?

    - Josiah
    Josiah Carlson, Oct 15, 2004
    #4
  5. On Fri, 15 Oct 2004 15:59:12 -0400, Darren Dale wrote:

    > I am doing linear algebra with large numarray. It is very efficient, but I
    > have a small problem due to the size of my data. The dot product of a
    > 10,000x3 double array with a 3x6,250,000 double array" will consume
    > 500GB of memory. I need to break the operations up into managable
    > chunks, so I dont consume all the available memory and get a
    > segmentation fault.
    >
    > Its not a problem with numpy, I just need to intelligently slice up one
    > of my arrays so my routine works within the available system resources.
    > Are there any utilities that can query how much memory is available?


    I don't know what you're doing with that, but you're well into the domain
    where you may have to trade running time for memory.

    I am not familiar with the terms "10,000x3 double array with a 3x6,250,000
    double array" (particularly "double array"), but speaking in general
    terms, assuming the dot product is something like the vector dot product I
    know, you can wrap your two source arrays in an object that lazily
    computes the relevant dot product. Shell:

    class LazyDotProduct(object):
    def __init__(self, a, b):
    self.a = a
    self.b = b

    def __getitem__(self, index):
    return dot_prodect(self.a, self.b, index)

    Add an optional cache to getitem if you need it and can afford it.
    "dot_product" computes the relevant dot product element.

    Just a thought; I may be over-extrapolating from what I know.
    Jeremy Bowers, Oct 16, 2004
    #5
  6. On Fri, 15 Oct 2004 23:29:48 +0000, Jeremy Bowers wrote:

    > I am not familiar with the terms "10,000x3 double array with a 3x6,250,000
    > double array" (particularly "double array"),


    Oh, duh, array of "doubles". The specification of dimensions had me
    thinking of some sort of array where each cell had 2 elements in it or
    something :)

    Now I am pretty sure you can compute it lazily.
    Jeremy Bowers, Oct 16, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lionel B
    Replies:
    2
    Views:
    376
    Lionel B
    Apr 23, 2005
  2. rtheiss

    Available System Memory

    rtheiss, Nov 6, 2003, in forum: Python
    Replies:
    2
    Views:
    436
    Tim Roberts
    Nov 9, 2003
  3. Replies:
    1
    Views:
    785
    Sebastian Millies
    Nov 2, 2006
  4. markus
    Replies:
    22
    Views:
    638
    Dances With Crows
    Sep 22, 2004
  5. Ángel José Riesgo
    Replies:
    13
    Views:
    477
    Ángel José Riesgo
    Feb 18, 2011
Loading...

Share This Page