Advice on persistent storage in Java?

[john martin]

I need some advice on persistent storage in Java. Basically, I'm writing
an application that I'd like to be in pure Java (including any backend
stuff it relies on, I'd like it to all run in the same VM). It has to
store a potentially large data structure. I'd like to have as little
hassle with SQL as possible, since I haven't done a ton of DB stuff,
though it seems that it may be necessary, and so I'm not totally averse
to using a regular SQL DB.

The two general options I'm considering are using an object oriented DB
(e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).

Both PERST and hsqldb are pure Java, so that meets my first requirement
(basically, I don't want someone to have to install a seperate DB
product to run my app). I like the idea of using an OODB, but it's not
totally transparent, and I won't be doing the most complex SQL in the
world, so I'm not sure if it'll really save me any work. Has anyone had
experience with either of the above tools? Anyone have any general
advice for easily storing (possibly large) object oriented data
structures in a way that's rather efficient and fault tolerant (i.e.,
not a flat text or XML file) but not a complete pain in the ass to
implement?

Any advice from people who've either done object oriented DB stuff or
who've had to bundle basic DB capability in a Java application would be
much appreciated.

-john

 

[Dimitri Maziuk]

john martin sez:

I need some advice on persistent storage in Java. ....
The two general options I'm considering are using an object oriented DB
(e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).

I've never used PERST & I use HSQLDB all the time.

Is your data structure an object? -- i.e. does it have to
have methods? If it's just data, it should map to regular
relational schema, no need for OO DB. (You can store objects
in HSQLDB, BTW, I just never needed that myself.)

Most database engines are optimized for query speed. If you
need to repeatedly load your data into DB, you won't get good
performance (i.e. lots of INSERT/UPDATEs and relatively few
SELECTs vs. lots of SELECTs & few INSERTs).

HSQLDB doesn't have a good query optimizer (how much smarts can
you fit into 200Kb?) so if you use a lot of complex SQL you may
not get good performance.

What you do get from RDB is SQL. From PERST docs, join of two
tables requires half a dozen lines of code and as many objects
(indexes, projections, result array, comparator). In SQL it's
one line: less code to write, lower development & maintenance
costs.

OTGH, if you don't need advanced search capabilities of SQL
and can do all your access via Hashmap or two, why not just
make your data Serializable and dump it to disk yourself?

Dima

 

[kjc]

Use hibernate http://www.hibernate.org

john martin sez:



I've never used PERST & I use HSQLDB all the time.

Is your data structure an object? -- i.e. does it have to
have methods? If it's just data, it should map to regular
relational schema, no need for OO DB. (You can store objects
in HSQLDB, BTW, I just never needed that myself.)

Most database engines are optimized for query speed. If you
need to repeatedly load your data into DB, you won't get good
performance (i.e. lots of INSERT/UPDATEs and relatively few
SELECTs vs. lots of SELECTs & few INSERTs).

HSQLDB doesn't have a good query optimizer (how much smarts can
you fit into 200Kb?) so if you use a lot of complex SQL you may
not get good performance.

What you do get from RDB is SQL. From PERST docs, join of two
tables requires half a dozen lines of code and as many objects
(indexes, projections, result array, comparator). In SQL it's
one line: less code to write, lower development & maintenance
costs.

OTGH, if you don't need advanced search capabilities of SQL
and can do all your access via Hashmap or two, why not just
make your data Serializable and dump it to disk yourself?

Dima

 

[john martin]

john martin sez:



I've never used PERST & I use HSQLDB all the time.

Is your data structure an object? -- i.e. does it have to
have methods? If it's just data, it should map to regular
relational schema, no need for OO DB. (You can store objects
in HSQLDB, BTW, I just never needed that myself.)

the structure is a couple of different types of objects, although the
objects are meant to store user defined data (the user defines the
structure of the data in addition to the values), so the schema wouldn't
be known ahead of time. so it seems easiest to be able to store whole
objects.

Most database engines are optimized for query speed. If you
need to repeatedly load your data into DB, you won't get good
performance (i.e. lots of INSERT/UPDATEs and relatively few
SELECTs vs. lots of SELECTs & few INSERTs).

yeah, it'd be more querying than updating.

HSQLDB doesn't have a good query optimizer (how much smarts can
you fit into 200Kb?) so if you use a lot of complex SQL you may
not get good performance.

the SQL would probably all be pretty basic.

What you do get from RDB is SQL. From PERST docs, join of two
tables requires half a dozen lines of code and as many objects
(indexes, projections, result array, comparator). In SQL it's
one line: less code to write, lower development & maintenance
costs.

OTGH, if you don't need advanced search capabilities of SQL
and can do all your access via Hashmap or two, why not just
make your data Serializable and dump it to disk yourself?

i guess i'm a bit wary of using serialization for long term storage, but
probably overly so. i'm thinking about using serialization for now
while i get started, and then switching over to something more robust
later on. although if i end up using hibernate, which lots of people
have recommended (i posted this same question in a couple of other
places), that wouldn't be the best design decision, since it would
ideally be more tightly integrated into the code than using
serialization or a DB via a persistence layer.

 

[Martin Egholm Nielsen]

Hi,

I need some advice on persistent storage in Java. === 8< 8< 8< ===
Anyone have any general advice for easily storing (possibly large)
object oriented data structures in a way that's rather efficient and
fault tolerant (i.e., not a flat text or XML file) but not a complete
pain in the ass to implement?

I know you write "not a flat XML file" - but even then, as a last way
out, look at http://xstream.codehaus.org/.

BR,
Martin Egholm

 

[Yamin]

heh,

1. Are you worried about runtime memory (that is you don't actually
want to load all the data into your program at once)?
2. Are you worried about the size of the persistane storage?
3. Do you need search abilities that aren't part of your data objects?

If you answer yes to 1, then a database is probably your only choice.
I'd really only worry about this, if your data starts running into the
10s/100s of megabytes.

How about the standard java XMLEncoder class (I think its in the beans
package) as an option even though you don't want XLM or flat file. Why
XMLEncoder/decoder? ...because its really easy to use, even though you
don't actually want to use it

As long as all the object you want to store:
1. have default constructors
2. have get/set in standard form for all members

you should be okay. You can perform custom delegates and stuff, but
for the most part, the two rules above are plenty.

The big advantage of XMLEncoder is that you can add/delete members
without dropping file support. I've had to use this 'feature' many
many many times.

Suppose in version1 of class ABC, there are 3 member variables m1, m2,
m3. I save several XMLencoder files in under version1. In version2, I
add member variable m4, and remove m2. A version2 program can still
read a version1 file.

The downside is of course a larger file size. If you really wanted to
be cool, you could zip the file as well, and unzip on load
(java.util.zip). This way, you get the ease of XML encoder, with
little persistance storage.