too much memory use

R

rohit

hi ,
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)
please help..
rohit sethi
delhi college of engineering
 
M

Marc 'BlackJack' Rintsch

i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)

I'm not quite sure what exactly you have as in-memory data structures and
how many "records" -- are you sure you don't keep references to objects
you don't really need anymore? Or maybe you have object cycles and
implemented the `__del__()` method on those objects?

Anyway… If the data doesn't fit into memory anymore it's time to put
them into a database. Either a `shelve`, an SQL database like SQLite or
maybe an object database like zodb or Durus.

Ciao,
Marc 'BlackJack' Rintsch
 
T

Tim Roberts

rohit said:
i am making a program for desktop search.
in order to make it more effective im implementing the records in the
form of a tree(files starting with 'a','b','c'....have different
trees ..so 26 trees in all) in memory and writing it down in file.
the max size file has 16000 records...now to implement the tree in
list i'm using line no as index ..and empty child nodes are
represented as "\n"
all this work is going on in the memory..
problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
cant think of an effective way to implement tree in memory(i can
compact it on disk by writing just the index no..along with the record
from which tree in memory can be reconstructed, but i have to
implement tree as previous to implement random access)

Why don't you just use a database? That's what they're designed for.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top