python memory blow out

[Simon Wittber]

I have written some software which proxy's SQL Server database
services across a network. It uses Pyro, without multiuthreading. It
creates and closes a new connection and cursor object for each
request.

Unfortunately, the memory consumption blows out (consuming all
available memory) when a large SQL query is run. This is not a problem
in itself; the problem is that Python does not release this memory
back to the operating system once it has finished with the result set.

I've noticed others have been frustrated by this problem also:
http://mail.python.org/pipermail/python-dev/2004-October/049483.html

According to the above post:

a) If the allocation is > 256 bytes, call the system malloc.
b) If the allocation is < 256, use its own malloc implementation, which
allocates memory in 256 kB chunks and never releases it.

I imagine this means that large memory allocations are eventually
released back to the operating system. However, in my case, this
appears to be not happening.

Does anyone have ideas on why this is occuring, or how I might
otherwise prevent memory blow out?

Sw.

 

[Istvan Albert]

Simon Wittber wrote:

Does anyone have ideas on why this is occuring, or how I might
otherwise prevent memory blow out?

I don't know it this is a decent enough solution but
if I were you I would try running the SQL service
in a subshell. Within this subshell I would
terminate then restart the program after a large
query and keep it going after small ones.

The clients then would need to have some tolerance
for the service being offline for short periods.

Just a guess.

Istvan.