- Re: Data/File Structure and Algorithm for Retrieving Sorted Data Chunk Efficiently

[Jane Austine]

"Fairly long enough"? I imagine you'd have to have thousands of
students before just doing the simplest thing (ignore performance
and read the whole list) would become a noticable problem.

First of all, thank you for replying me.

Yes, I do have more than a thousand.

It's rare in an environment like the web, with all the user and
network delays that are present, that optimizing at the level
you are trying to do is worth the effort.

Yes. Especially when there are CPU load limits(process going on like a
few seconds using more than 10% will get warning) on the server, and
it is shared among a few users(using a web hosting service).

And unless mod_python is involved, CGI adds yet another significant
delay, masking any slight difference in performance from the optimal
approach and the simplistic one.

True. So the situation is not very good, and that's why I need a
clever scheme.

Nope, simplest would be just use a list and ignore the theoretical
but unnoticable performance issues. Remember, the actual memory
allocation and manipulations of the list elements happens in C code,
not in Python, so it is very fast.


If you've got more than 1000 students, it _might_ be worth
analyzing this further.

Actually, it's not exactly students. They are some kind of elements
and they have a few attributes(columns) like timestamp and etc, and
can be sorted by a few columns(single key sorting though).

Most of the time I suppose the lists would grow up to 10000 ~ 90000
elements.

-Jane

 

[Changjune Kim]

First of all, thank you for replying me.

Yes, I do have more than a thousand.


Yes. Especially when there are CPU load limits(process going on like a
few seconds using more than 10% will get warning) on the server, and
it is shared among a few users(using a web hosting service).


True. So the situation is not very good, and that's why I need a
clever scheme.


Actually, it's not exactly students. They are some kind of elements
and they have a few attributes(columns) like timestamp and etc, and
can be sorted by a few columns(single key sorting though).

Most of the time I suppose the lists would grow up to 10000 ~ 90000
elements.

-Jane

If using standalone RDBMS engine(like MySQL) is not your choice, and you
want to stick to bsddb, take a look at test_basics.py in the lib/bsddb/test
directory. There you will see BTreeRecnoTestCase.

 

[Changjune Kim]

First of all, thank you for replying me.

Yes, I do have more than a thousand.


Yes. Especially when there are CPU load limits(process going on like a
few seconds using more than 10% will get warning) on the server, and
it is shared among a few users(using a web hosting service).


True. So the situation is not very good, and that's why I need a
clever scheme.


Actually, it's not exactly students. They are some kind of elements
and they have a few attributes(columns) like timestamp and etc, and
can be sorted by a few columns(single key sorting though).

Most of the time I suppose the lists would grow up to 10000 ~ 90000
elements.

-Jane

If using standalone RDBMS engine(like MySQL) is not your choice, and you
want to stick to bsddb, take a look at test_basics.py in the lib/bsddb/test
directory. There you will see BTreeRecnoTestCase.