limiting memory consumption of Python itself or a dict in a Python program

J

Jonas Maurus

Hello everybody,

I'm pondering the following problem:

I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Is this possible? How would I go about implementing it? By imposing
"ulimit"s and catching MemoryError?

Thanks for your help,
Jonas
 
K

Kay Schluehr

Hello everybody,

I'm pondering the following problem:

I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Is this possible? How would I go about implementing it? By imposing
"ulimit"s and catching MemoryError?

Thanks for your help,
Jonas

When your application pre-allocates a chunk of memory and use
datatypes like array and struct you might write your own memory
manager. You can't control the total memory consumption of Python
though, since Python implements an own malloc and it might allocate
memory in large chunks. But this is somewhat orthogonal to your
problem and applies also when Python reads/writes buffers from/to
disk.
 
J

Jeremy Sanders

Jonas said:
I want to write a Python program that receives messages via SMTP and
stores them in a dict or an array. For my purposes it would be
important that all received mail would be kept in RAM and not cached
out to disk. If a new message comes in that can't fit in the allocated
memory, a number of old messages would be discarded.

As the server needs to have room for other tasks, I'd like to limit
the overall memory consumption to a certain amount.

Since your data is all in one place, why not write a dict or list wrapper
which keeps track of the total space of the items stored (using len on the
strings)?

It could automatically clean out old entries when the memory usage becomes
too much.

Jeremy
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top