Advice on persistent storage in Java?

Discussion in 'Java' started by john martin, Dec 11, 2004.

  1. john martin

    john martin Guest

    I need some advice on persistent storage in Java. Basically, I'm writing
    an application that I'd like to be in pure Java (including any backend
    stuff it relies on, I'd like it to all run in the same VM). It has to
    store a potentially large data structure. I'd like to have as little
    hassle with SQL as possible, since I haven't done a ton of DB stuff,
    though it seems that it may be necessary, and so I'm not totally averse
    to using a regular SQL DB.

    The two general options I'm considering are using an object oriented DB
    (e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
    SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).

    Both PERST and hsqldb are pure Java, so that meets my first requirement
    (basically, I don't want someone to have to install a seperate DB
    product to run my app). I like the idea of using an OODB, but it's not
    totally transparent, and I won't be doing the most complex SQL in the
    world, so I'm not sure if it'll really save me any work. Has anyone had
    experience with either of the above tools? Anyone have any general
    advice for easily storing (possibly large) object oriented data
    structures in a way that's rather efficient and fault tolerant (i.e.,
    not a flat text or XML file) but not a complete pain in the ass to
    implement?

    Any advice from people who've either done object oriented DB stuff or
    who've had to bundle basic DB capability in a Java application would be
    much appreciated.

    -john
     
    john martin, Dec 11, 2004
    #1
    1. Advertising

  2. john martin sez:
    > I need some advice on persistent storage in Java.

    ....
    > The two general options I'm considering are using an object oriented DB
    > (e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
    > SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).


    I've never used PERST & I use HSQLDB all the time.

    Is your data structure an object? -- i.e. does it have to
    have methods? If it's just data, it should map to regular
    relational schema, no need for OO DB. (You can store objects
    in HSQLDB, BTW, I just never needed that myself.)

    Most database engines are optimized for query speed. If you
    need to repeatedly load your data into DB, you won't get good
    performance (i.e. lots of INSERT/UPDATEs and relatively few
    SELECTs vs. lots of SELECTs & few INSERTs).

    HSQLDB doesn't have a good query optimizer (how much smarts can
    you fit into 200Kb?) so if you use a lot of complex SQL you may
    not get good performance.

    What you do get from RDB is SQL. From PERST docs, join of two
    tables requires half a dozen lines of code and as many objects
    (indexes, projections, result array, comparator). In SQL it's
    one line: less code to write, lower development & maintenance
    costs.

    OTGH, if you don't need advanced search capabilities of SQL
    and can do all your access via Hashmap or two, why not just
    make your data Serializable and dump it to disk yourself?

    Dima
    --
    True courage comes from steadying yourself and forcing yourself to ssh into the
    fscking thing yet again and not admitting that it doesn't care what it's done
    to your life. -- "Hidden among the nodes" by ADB
     
    Dimitri Maziuk, Dec 11, 2004
    #2
    1. Advertising

  3. john martin

    kjc Guest

    Use hibernate http://www.hibernate.org

    Dimitri Maziuk wrote:
    > john martin sez:
    >
    >>I need some advice on persistent storage in Java.

    >
    > ...
    >
    >>The two general options I'm considering are using an object oriented DB
    >>(e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
    >>SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).

    >
    >
    > I've never used PERST & I use HSQLDB all the time.
    >
    > Is your data structure an object? -- i.e. does it have to
    > have methods? If it's just data, it should map to regular
    > relational schema, no need for OO DB. (You can store objects
    > in HSQLDB, BTW, I just never needed that myself.)
    >
    > Most database engines are optimized for query speed. If you
    > need to repeatedly load your data into DB, you won't get good
    > performance (i.e. lots of INSERT/UPDATEs and relatively few
    > SELECTs vs. lots of SELECTs & few INSERTs).
    >
    > HSQLDB doesn't have a good query optimizer (how much smarts can
    > you fit into 200Kb?) so if you use a lot of complex SQL you may
    > not get good performance.
    >
    > What you do get from RDB is SQL. From PERST docs, join of two
    > tables requires half a dozen lines of code and as many objects
    > (indexes, projections, result array, comparator). In SQL it's
    > one line: less code to write, lower development & maintenance
    > costs.
    >
    > OTGH, if you don't need advanced search capabilities of SQL
    > and can do all your access via Hashmap or two, why not just
    > make your data Serializable and dump it to disk yourself?
    >
    > Dima
     
    kjc, Dec 11, 2004
    #3
  4. john martin

    john martin Guest

    Dimitri Maziuk wrote:
    > john martin sez:
    >
    >>I need some advice on persistent storage in Java.

    >
    > ...
    >
    >>The two general options I'm considering are using an object oriented DB
    >>(e.g., PERST, http://www.garret.ru/~knizhnik/perst.html), and a regular
    >>SQL DB (e.g., hsqldb, http://hsqldb.sourceforge.net/).

    >
    >
    > I've never used PERST & I use HSQLDB all the time.
    >
    > Is your data structure an object? -- i.e. does it have to
    > have methods? If it's just data, it should map to regular
    > relational schema, no need for OO DB. (You can store objects
    > in HSQLDB, BTW, I just never needed that myself.)


    the structure is a couple of different types of objects, although the
    objects are meant to store user defined data (the user defines the
    structure of the data in addition to the values), so the schema wouldn't
    be known ahead of time. so it seems easiest to be able to store whole
    objects.

    >
    > Most database engines are optimized for query speed. If you
    > need to repeatedly load your data into DB, you won't get good
    > performance (i.e. lots of INSERT/UPDATEs and relatively few
    > SELECTs vs. lots of SELECTs & few INSERTs).


    yeah, it'd be more querying than updating.

    >
    > HSQLDB doesn't have a good query optimizer (how much smarts can
    > you fit into 200Kb?) so if you use a lot of complex SQL you may
    > not get good performance.


    the SQL would probably all be pretty basic.

    >
    > What you do get from RDB is SQL. From PERST docs, join of two
    > tables requires half a dozen lines of code and as many objects
    > (indexes, projections, result array, comparator). In SQL it's
    > one line: less code to write, lower development & maintenance
    > costs.
    >
    > OTGH, if you don't need advanced search capabilities of SQL
    > and can do all your access via Hashmap or two, why not just
    > make your data Serializable and dump it to disk yourself?


    i guess i'm a bit wary of using serialization for long term storage, but
    probably overly so. i'm thinking about using serialization for now
    while i get started, and then switching over to something more robust
    later on. although if i end up using hibernate, which lots of people
    have recommended (i posted this same question in a couple of other
    places), that wouldn't be the best design decision, since it would
    ideally be more tightly integrated into the code than using
    serialization or a DB via a persistence layer.

    >
    > Dima
     
    john martin, Dec 13, 2004
    #4
  5. Hi,

    > I need some advice on persistent storage in Java.

    === 8< 8< 8< ===
    > Anyone have any general advice for easily storing (possibly large)
    > object oriented data structures in a way that's rather efficient and
    > fault tolerant (i.e., not a flat text or XML file) but not a complete
    > pain in the ass to implement?

    I know you write "not a flat XML file" - but even then, as a last way
    out, look at http://xstream.codehaus.org/.

    BR,
    Martin Egholm
     
    Martin Egholm Nielsen, Dec 15, 2004
    #5
  6. john martin

    Yamin Guest

    heh,

    1. Are you worried about runtime memory (that is you don't actually
    want to load all the data into your program at once)?
    2. Are you worried about the size of the persistane storage?
    3. Do you need search abilities that aren't part of your data objects?

    If you answer yes to 1, then a database is probably your only choice.
    I'd really only worry about this, if your data starts running into the
    10s/100s of megabytes.

    How about the standard java XMLEncoder class (I think its in the beans
    package) as an option even though you don't want XLM or flat file. Why
    XMLEncoder/decoder? ...because its really easy to use, even though you
    don't actually want to use it :)

    As long as all the object you want to store:
    1. have default constructors
    2. have get/set in standard form for all members

    you should be okay. You can perform custom delegates and stuff, but
    for the most part, the two rules above are plenty.

    The big advantage of XMLEncoder is that you can add/delete members
    without dropping file support. I've had to use this 'feature' many
    many many times.

    Suppose in version1 of class ABC, there are 3 member variables m1, m2,
    m3. I save several XMLencoder files in under version1. In version2, I
    add member variable m4, and remove m2. A version2 program can still
    read a version1 file.

    The downside is of course a larger file size. If you really wanted to
    be cool, you could zip the file as well, and unzip on load
    (java.util.zip). This way, you get the ease of XML encoder, with
    little persistance storage.
     
    Yamin, Dec 15, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Frank Gerlach

    Advice on persistent storage in Java?

    Frank Gerlach, Dec 12, 2004, in forum: Java
    Replies:
    0
    Views:
    378
    Frank Gerlach
    Dec 12, 2004
  2. Asun Friere
    Replies:
    1
    Views:
    520
    Paul Boddie
    Aug 27, 2003
  3. sarathy
    Replies:
    2
    Views:
    688
    sarathy
    Jul 17, 2006
  4. gk
    Replies:
    7
    Views:
    1,020
    Tom Anderson
    Oct 12, 2010
  5. Yan Huang
    Replies:
    4
    Views:
    131
    Thomas 'PointedEars' Lahn
    Oct 19, 2009
Loading...

Share This Page