limiting memory consumption of Python itself or a dict in a Python program

Discussion in 'Python' started by Jonas Maurus, Sep 19, 2007.

  1. Jonas Maurus

    Jonas Maurus Guest

    Hello everybody,

    I'm pondering the following problem:

    I want to write a Python program that receives messages via SMTP and
    stores them in a dict or an array. For my purposes it would be
    important that all received mail would be kept in RAM and not cached
    out to disk. If a new message comes in that can't fit in the allocated
    memory, a number of old messages would be discarded.

    As the server needs to have room for other tasks, I'd like to limit
    the overall memory consumption to a certain amount.

    Is this possible? How would I go about implementing it? By imposing
    "ulimit"s and catching MemoryError?

    Thanks for your help,
    Jonas
     
    Jonas Maurus, Sep 19, 2007
    #1
    1. Advertising

  2. Jonas Maurus

    Kay Schluehr Guest

    On 19 Sep., 22:48, Jonas Maurus <> wrote:
    > Hello everybody,
    >
    > I'm pondering the following problem:
    >
    > I want to write a Python program that receives messages via SMTP and
    > stores them in a dict or an array. For my purposes it would be
    > important that all received mail would be kept in RAM and not cached
    > out to disk. If a new message comes in that can't fit in the allocated
    > memory, a number of old messages would be discarded.
    >
    > As the server needs to have room for other tasks, I'd like to limit
    > the overall memory consumption to a certain amount.
    >
    > Is this possible? How would I go about implementing it? By imposing
    > "ulimit"s and catching MemoryError?
    >
    > Thanks for your help,
    > Jonas


    When your application pre-allocates a chunk of memory and use
    datatypes like array and struct you might write your own memory
    manager. You can't control the total memory consumption of Python
    though, since Python implements an own malloc and it might allocate
    memory in large chunks. But this is somewhat orthogonal to your
    problem and applies also when Python reads/writes buffers from/to
    disk.
     
    Kay Schluehr, Sep 20, 2007
    #2
    1. Advertising

  3. Jonas Maurus wrote:

    > I want to write a Python program that receives messages via SMTP and
    > stores them in a dict or an array. For my purposes it would be
    > important that all received mail would be kept in RAM and not cached
    > out to disk. If a new message comes in that can't fit in the allocated
    > memory, a number of old messages would be discarded.
    >
    > As the server needs to have room for other tasks, I'd like to limit
    > the overall memory consumption to a certain amount.


    Since your data is all in one place, why not write a dict or list wrapper
    which keeps track of the total space of the items stored (using len on the
    strings)?

    It could automatically clean out old entries when the memory usage becomes
    too much.

    Jeremy

    --
    Jeremy Sanders
    http://www.jeremysanders.net/
     
    Jeremy Sanders, Sep 20, 2007
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shahriar Shamil Uulu
    Replies:
    5
    Views:
    512
    Olivier Grisel
    Dec 24, 2005
  2. k3xji
    Replies:
    10
    Views:
    682
    Hrvoje Niksic
    Apr 10, 2009
  3. Replies:
    2
    Views:
    198
  4. Ethan Furman

    Python 3: dict & dict.keys()

    Ethan Furman, Jul 24, 2013, in forum: Python
    Replies:
    4
    Views:
    267
    Steven D'Aprano
    Jul 25, 2013
  5. Peter Otten

    Re: Python 3: dict & dict.keys()

    Peter Otten, Jul 24, 2013, in forum: Python
    Replies:
    1
    Views:
    113
    Neil Cerutti
    Jul 24, 2013
Loading...

Share This Page