To store a huge table during start-up of a J2EE application

[Ravi Shankar Nair]

Hi all,

Rather than making a db query all the time, I am considering the design of
stroing a huge table( with 12 columns and may be about 500 rows) in some
data structure , during start-up of my servlet. Memory is not a constraint,
thats why we hare going for such an implementation. Please help for some
questions below:-

1) Whats the best ADT or Data Structure for doing this?
2) In order to reduce the number of rows and ease manipulation, is it better
to have an HttpBindinngListener which will strone the info about the user
who had logged in and corresponding values/rows for that user?
3)Any other way ?

Thanks for your time and advice.

BEst regards,
Ravi

 

[Rhino]

Hi all,

Rather than making a db query all the time, I am considering the design of
stroing a huge table( with 12 columns and may be about 500 rows) in some
data structure , during start-up of my servlet. Memory is not a
constraint, thats why we hare going for such an implementation. Please
help for some questions below:-

1) Whats the best ADT or Data Structure for doing this?
2) In order to reduce the number of rows and ease manipulation, is it
better to have an HttpBindinngListener which will strone the info about
the user who had logged in and corresponding values/rows for that user?
3)Any other way ?

Thanks for your time and advice.

Just a quibble about the wording of your subject line and question: I
wouldn't call 500 rows of data with 12 columns of data in each row a "huge"
table. I'm a database guy and we don't usually think a table is "huge" until
it has MILLIONS of rows - or more. Now, if your columns are really wide, say
2 GB each or something like that, you are starting to get into slightly more
significant territory but, generally speaking, I wouldn't call your table
anything more than "small".

As for the best design for what you are doing, I can't really suggest a good
answer since I've never embedded a table of the kind you describe in a
servlet. My servlets always get that sort of data from the database.

Is this static data? In other words, is it ever going to change or is it
just a lookup table of some kind, like the two-letter codes for the
different states in the US? If the data is NOT static, I hope you give some
thought to how you are going to keep it consistent with the source of the
data, whether that is a database or a file. If the data can change, let's
say it's a price table, you're going to run into BIG problems if the servlet
version of the table has the old prices and you charge customers at the old
prices when the new prices have gone into effect.

 

[Ravi Shankar Nair]

Dear Rhino,

I agree, the table is small as per your thoughts. Though for us, it is abig
table.

Yah, basically it is alook up kind of data. FOr example the role
categories, the function ids, the action ids etc for a particular user. When
the user logs in, we need to check the entitlement, whether this user is
permitted to go for this function or not. Thats why we are caching it. Our
priority is not for memory concumption, but execution time. Thats why we
cant have a look up from servlet through an EJB or DAO kind of thing.

Any thoughts to share with? After contemplating a bit, what I am thinking is
this:- Assume that first four columns, col1...col4 are primary keys out of
the total 12 columns. My search or lookup is always based on the primary
key, i.e the first four columns. So I will have the startup init method in
one of the servlets, where I will create two strings. First one is
"col1__col2__col3__col4" (lets say the key)and second one is
"col5:col6:col7:col8:...col12" (lets say the value).Then I will populate a
hashtable with key and value.

Whenever a search is needed, I would first construct the key and look for
it.Once I get the value, I will STringTokenize it and use it wherever
required.

In case I need a search for only col1( may be in future), I would have
another hashtable to maintain col1_...col4.

Is this fine? Please comment. Thanks

Best regards,
Ravi

 

[Arvind]

Dear Rhino,

I agree, the table is small as per your thoughts. Though for us, it is abig
table.

Yah, basically it is alook up kind of data. FOr example the role
categories, the function ids, the action ids etc for a particular user. When
the user logs in, we need to check the entitlement, whether this user is
permitted to go for this function or not. Thats why we are caching it. Our
priority is not for memory concumption, but execution time. Thats why we
cant have a look up from servlet through an EJB or DAO kind of thing.

Any thoughts to share with? After contemplating a bit, what I am thinking is
this:- Assume that first four columns, col1...col4 are primary keys out of
the total 12 columns. My search or lookup is always based on the primary
key, i.e the first four columns. So I will have the startup init method in
one of the servlets, where I will create two strings. First one is
"col1__col2__col3__col4" (lets say the key)and second one is
"col5:col6:col7:col8:...col12" (lets say the value).Then I will populate a
hashtable with key and value.

Whenever a search is needed, I would first construct the key and look for
it.Once I get the value, I will STringTokenize it and use it wherever
required.

In case I need a search for only col1( may be in future), I would have
another hashtable to maintain col1_...col4.

Is this fine? Please comment. Thanks

The first thing to do is to create an Object that encapsulates these 12
columns i.e. attributes of that row.

Second this is (assuming for a second that your keying col1...col4 key
is good enough), store the "object" that you created in the previous
step as value of the hashtable - dont get into the business of
stringtokenizing etc, they are costly and round-about way of doing the
things at hand.

Now to your actual problem, if your primary keys and index are setup
correctly, a database query should not run for more than tens of
milliseconds - granted that there are some exceptions too. Since you
seem to indicate user-specific data, dont see the value of losing the
ability to do SQL like queries in return for 10 millis of time. And
actually, when you scale up with number of users, the memory will kinda
get abused, but the database can easily scale up to a high concurrent
query load.

 

[Patricia Shanahan]

Ravi Shankar Nair wrote:
....

Any thoughts to share with? After contemplating a bit, what I am thinking is
this:- Assume that first four columns, col1...col4 are primary keys out of
the total 12 columns. My search or lookup is always based on the primary
key, i.e the first four columns. So I will have the startup init method in
one of the servlets, where I will create two strings. First one is
"col1__col2__col3__col4" (lets say the key)and second one is
"col5:col6:col7:col8:...col12" (lets say the value).Then I will populate a
hashtable with key and value.

....

I don't recommend encoding as String and tokenizing. It makes everything
quite unnecessarily complicated. You would need to either impose a
prohibition on "_" as data, or do some quoting.

It would be simpler to create classes for the keys and values. You would
need to write good, consistent, equals and hashCode methods for the key
class.

Also, don't ignore the database idea. It scales to almost any size, and
minimizes the amount of change in your program that is required to
support additional look-up methods.

Patricia

 

[Domagoj Klepac]

Rather than making a db query all the time, I am considering the design of
stroing a huge table( with 12 columns and may be about 500 rows) in some
data structure , during start-up of my servlet. Memory is not a constraint,
thats why we hare going for such an implementation. Please help for some
questions below:-

1) Whats the best ADT or Data Structure for doing this?

Typically, that's one collection (for table) stored inside the other
collection (for record).

But it really depends on what you need to do with that table.

If you're always fetching a record on several criteria, you might want
to use List inside the List.

If you have one key column, and are always accessing data through that
column? That's Map, where you store key column in key, and List in
value.

Or if your access patterns are more complicated, you might want to
roll your own classes. Use one class with getters and setters for
record, and than store that records in some sort of list. Or in other
custom class which uses list to store records, but has custom methods
which access the list.

If your table is not read-only, you might want to have custom
add/put/set method which writes record to both in-memory table and to
the database. Or have some sort of queue where you're marking which
records were added or need updating, and fork another thread to empty
that queue.

If several threads need to access the table, watch for thread safety.

Domchi

 

[David Segall]

Hi all,

Rather than making a db query all the time, I am considering the design of
stroing a huge table( with 12 columns and may be about 500 rows) in some
data structure , during start-up of my servlet. Memory is not a constraint,
thats why we hare going for such an implementation. Please help for some
questions below:-

1) Whats the best ADT or Data Structure for doing this?
2) In order to reduce the number of rows and ease manipulation, is it better
to have an HttpBindinngListener which will strone the info about the user
who had logged in and corresponding values/rows for that user?
3)Any other way ?

Perhaps you can have the best of both worlds by using an in-memory
database such as <http://hsqldb.org/>.