Java class persistance

[kt]

I'm developing an application using Java and I would like the user to
be able to plug in various methods of persistance. The has led me to
JDBC using standard SQL-92, but this is really overkill for what I'm
trying to do. I only have a handful of classes that I would like to
persist and I won't be doing any complicated querying on them. I will
always be looking up objects by key values. In addition I would like
to be able to ship a default file based back end with the code for
users that aren't interested in the extra hassle of setting up their
own, more sophisticated, persistance.

I looked at JDO briefly and it doesn't seem to allow what I want. It
looks like the programmer must select the database back end at
development time. Is this correct?

I also thought about using object serialization in some way to achieve
the objective, but this doesn't work. I need to be able to have Maps
that contain more objects than can fit in memory at any given time.

Any suggestions would be appreciated. Thanks.

 

[Antti S. Brax]

I also thought about using object serialization in some way to achieve
the objective, but this doesn't work. I need to be able to have Maps
that contain more objects than can fit in memory at any given time.

Does the fact that you need "larger than memory" Maps prevent
the use of object serialization? How exactly? I can't see how
you could have the default Map implementations only partially
loaded. You will need a specialized implementation. And once
you do that you can just as well use object serialization for
persistence.

 

[Lucy]

Does the fact that you need "larger than memory" Maps prevent
the use of object serialization? How exactly? I can't see how
you could have the default Map implementations only partially
loaded. You will need a specialized implementation. And once
you do that you can just as well use object serialization for
persistence.

How do you make a map that is bigger than memory in the first place?
Don't you get an out of memory error?

 

[Lucy]

I'm developing an application using Java and I would like the user to
be able to plug in various methods of persistance. The has led me to
JDBC using standard SQL-92, but this is really overkill for what I'm
trying to do. I only have a handful of classes that I would like to
persist and I won't be doing any complicated querying on them. I will
always be looking up objects by key values. In addition I would like
to be able to ship a default file based back end with the code for
users that aren't interested in the extra hassle of setting up their
own, more sophisticated, persistance.

I looked at JDO briefly and it doesn't seem to allow what I want. It
looks like the programmer must select the database back end at
development time. Is this correct?

I also thought about using object serialization in some way to achieve
the objective, but this doesn't work. I need to be able to have Maps
that contain more objects than can fit in memory at any given time.

Any suggestions would be appreciated. Thanks.

How about a flat file. Each line could represent a tuple with the first N
characters being the key.

 

[Kenneth P. Turvey]

Does the fact that you need "larger than memory" Maps prevent
the use of object serialization? How exactly? I can't see how
you could have the default Map implementations only partially
loaded. You will need a specialized implementation. And once
you do that you can just as well use object serialization for
persistence.

The problem here is lookup speed. I could use object serialization for
lookups, but I would really be implementing the method of persistence
then. I would have to build indexes and other details having to do
with how the class is stored to disk. I'm really looking for a
solution that allows the user to plug in different methods of
persistence depending on how fast they need it to run, and how much
money they have to spend. I don't know if something general exists,
other than JDBC with standard SQL. I'm not really familiar with JDO so
I don't know if this technology has a way to solve this problem.
Certainly other programmers have had a need for generalized persistance
mechanism that doesn't tie them to a specific implementation.

 

[Kenneth P. Turvey]

The standard maps wouldn't work, but it is easy enough to implement one
with SoftReferences that acts as a cache to a persistance layer.

 

[Kenneth P. Turvey]

This has the same problem as serialization. It ties the end user to a
specific implementation of the persistence layer that may not be
optimal for what they want to do. It would work, but there are several
solutions that would solve the problem without giving the end user the
flexibility I would like to.

 

[Lucy]

In addition I would like

to be able to ship a default file based back end with the code for
users that aren't interested in the extra hassle of setting up their
own, more sophisticated, persistance.

what is in the file then
does it have to be read by every plugin they can choose from

 

[Kenneth P. Turvey]

what is in the file then
does it have to be read by every plugin they can choose from

No. I was thinking of some default plugin. Again, I'm just looking
for an existing technology that allows a generalized persistence layer.
I'm getting the impressing that JDBC with SQL-92 is really the only
way to go here.

 

[CodeFutures]

I'm developing an application using Java and I would like the user to
be able to plug in various methods of persistance.

There's generally too much attention paid to the choice of Java
persistence technology (e.g. Hibernate versus JDO) and not enough on
the overall architectural strategy.
Java Persistence Architecture - this should give you some ideas about
how to design a persistence architecture that allows you to exchange
different Java persistence technologies:

http://www.codefutures.com/weblog/andygrove/archives/2005/05/java_persistenc.html



There's a blog posting with a comparison of these Java persistence
technology choices here:

http://www.codefutures.com/weblog/corporate/archives/2005/02/data_persistenc.html

If you're looking for general advice on data persistence, you might
like to read this blog posting on choosing a choosing a data
persistence strategy:

http://www.codefutures.com/weblog/andygrove/archives/2005/01/choosing_a_java.html


This might be useful:

http://www.codefutures.com/weblog/andygrove/archives/2005/02/data_access_obj.html

PJ Murray
CodeFutures Software

 

[Antti S. Brax]

The problem here is lookup speed. I could use object serialization for
lookups, but I would really be implementing the method of persistence
then.

Can't you plug your selected persistence method to the indexes too?

 

[Ross Bamford]

I'm developing an application using Java and I would like the user to
be able to plug in various methods of persistance. The has led me to
JDBC using standard SQL-92, but this is really overkill for what I'm
trying to do. I only have a handful of classes that I would like to
persist and I won't be doing any complicated querying on them. I will
always be looking up objects by key values. In addition I would like
to be able to ship a default file based back end with the code for
users that aren't interested in the extra hassle of setting up their
own, more sophisticated, persistance.

I looked at JDO briefly and it doesn't seem to allow what I want. It
looks like the programmer must select the database back end at
development time. Is this correct?

I also thought about using object serialization in some way to achieve
the objective, but this doesn't work. I need to be able to have Maps
that contain more objects than can fit in memory at any given time.

Any suggestions would be appreciated. Thanks.

You're at the top of a deep rabbit hole, exactly where I was on
something similar a little while ago. I though 'well, it's only simple,
and I'm only fetching on key' and all that stuff.

I ended up with straight JDBC, a Serializable 'Stub' object, arbitrary
extra fields for the large data (which the objects transparently mapped
to properties) and so on. It worked, it was fast, but it was going
nowhere and I dumped it for Hibernate.

However simple you make it in your mind, you'll end up solving (hard)
problems that have already been solved. It's just that the existing
solutions are so damn hungry for metadata...