Java DB, is it for small databases only?

Discussion in 'Java' started by Lionel van den Berg, Mar 3, 2008.

  1. I'm looking into Java DB, and although Sun claims that it can be a full-
    scale DBMS I'm wondering what the views on this are? Can Java DB be used
    for large databases?

    Thanks

    Lionel.
     
    Lionel van den Berg, Mar 3, 2008
    #1
    1. Advertising

  2. Lionel van den Berg, 03.03.2008 11:19:
    > I'm looking into Java DB, and although Sun claims that it can be a full-
    > scale DBMS I'm wondering what the views on this are? Can Java DB be used
    > for large databases?


    Define "large"
     
    Thomas Kellerer, Mar 3, 2008
    #2
    1. Advertising

  3. On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:

    > Lionel van den Berg, 03.03.2008 11:19:
    >> I'm looking into Java DB, and although Sun claims that it can be a
    >> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >> be used for large databases?

    >
    > Define "large"


    Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
    do enough to get by.

    Let's say 10 tables with an average of 4 fields with each table having up
    to 10 million entries.

    Not sure if that constitutes large, I'm sure it's not to google.
     
    Lionel van den Berg, Mar 3, 2008
    #3
  4. Lionel van den Berg, 03.03.2008 12:47:
    >>> I'm looking into Java DB, and although Sun claims that it can be a
    >>> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >>> be used for large databases?

    >> Define "large"

    >
    > Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
    > do enough to get by.
    >
    > Let's say 10 tables with an average of 4 fields with each table having up
    > to 10 million entries.
    >
    > Not sure if that constitutes large, I'm sure it's not to google.


    OK, I would think that qualifies as "large" (at least not as small :) )

    I have no experience with Derby using such a setup. I would be a bit hesitating about using it for such a database. But I guess this can only be answered by setting up a testdatabase of that size.

    You might want to post that question to the Derby mailing list, support over there is quite good.

    Thomas
     
    Thomas Kellerer, Mar 3, 2008
    #4
  5. Lionel van den Berg wrote:
    > On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:
    >> Lionel van den Berg, 03.03.2008 11:19:
    >>> I'm looking into Java DB, and although Sun claims that it can be a
    >>> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >>> be used for large databases?

    >> Define "large"

    >
    > Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
    > do enough to get by.
    >
    > Let's say 10 tables with an average of 4 fields with each table having up
    > to 10 million entries.


    If it is integer fields then 10*4*10M = 400 MB which is a small
    database.

    Even though I have not tested Derby with a 400 MB database, then I
    am sure it can handle it.

    Arne
     
    Arne Vajhøj, Mar 3, 2008
    #5
  6. Arne Vajhøj wrote:
    > Lionel van den Berg wrote:
    >> On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:
    >>> Lionel van den Berg, 03.03.2008 11:19:
    >>>> I'm looking into Java DB, and although Sun claims that it can be a
    >>>> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >>>> be used for large databases?
    >>> Define "large"

    >>
    >> Sigh, I knew that was coming. I'm not sure, I'm not a database person,
    >> I do enough to get by.
    >>
    >> Let's say 10 tables with an average of 4 fields with each table having
    >> up to 10 million entries.

    >
    > If it is integer fields then 10*4*10M = 400 MB which is a small
    > database.
    >
    > Even though I have not tested Derby with a 400 MB database, then I
    > am sure it can handle it.


    Translation:

    MB = small
    GB = medium
    TB = large
    PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

    Arne
     
    Arne Vajhøj, Mar 3, 2008
    #6
  7. Lionel van den Berg

    Daniel Pitts Guest

    Arne Vajhøj wrote:
    >> Even though I have not tested Derby with a 400 MB database, then I
    >> am sure it can handle it.

    >
    > Translation:
    >
    > MB = small
    > GB = medium
    > TB = large
    > PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

    You mean: PB = Google

    --
    Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
     
    Daniel Pitts, Mar 3, 2008
    #7
  8. Arne Vajhøj wrote on 03.03.2008 22:46:
    > Lionel van den Berg wrote:
    >> On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:
    >>> Lionel van den Berg, 03.03.2008 11:19:
    >>>> I'm looking into Java DB, and although Sun claims that it can be a
    >>>> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >>>> be used for large databases?
    >>> Define "large"

    >>
    >> Sigh, I knew that was coming. I'm not sure, I'm not a database person,
    >> I do enough to get by.
    >>
    >> Let's say 10 tables with an average of 4 fields with each table having
    >> up to 10 million entries.

    >
    > If it is integer fields then 10*4*10M = 400 MB which is a small
    > database.
    >

    The actual data size is not necessarily an indicator whether a database is
    "large". 10 Million rows (per table) could be a problem, if e.g. the access path
    to retrieve a subset is not well chosen by the engine and it has to scan all
    rows in order to retrieve e.g. 5 of them.

    If the number of updates and parallel reads is very high, but the design does
    not allow for concurrent updates and reads ("writers blocking readers") then the
    performance of the application will suffer as.
    Btw: writers blocking readers is a problem even for some of the "big" names

    Thomas
     
    Thomas Kellerer, Mar 3, 2008
    #8
  9. Lionel van den Berg

    Roedy Green Guest

    On 3 Mar 2008 21:19:08 +1100, Lionel van den Berg <>
    wrote, quoted or indirectly quoted someone who said :

    >I'm looking into Java DB, and although Sun claims that it can be a full-
    >scale DBMS I'm wondering what the views on this are? Can Java DB be used
    >for large databases?


    Java DB (aka Derby, aka Cloudscape) is for small databases. MySQL
    which Sun now owns is for intermediate sized ones.

    For alternatives running the gamut, see
    http://mindprod.com/jgloss/sqlvendors.html

    --

    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
     
    Roedy Green, Mar 3, 2008
    #9
  10. Lionel van den Berg

    Christian Guest

    Daniel Pitts schrieb:
    > Arne Vajhøj wrote:
    >>> Even though I have not tested Derby with a 400 MB database, then I
    >>> am sure it can handle it.

    >>
    >> Translation:
    >>
    >> MB = small
    >> GB = medium
    >> TB = large
    >> PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

    > You mean: PB = Google
    >

    nope not Really.. I googled a and found some blog entry from 2006
    estimating Googles data in compressed form to 220TB. I doubt it grew to
    a PB since then.

    http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html
     
    Christian, Mar 4, 2008
    #10
  11. Lionel van den Berg

    Daniel Pitts Guest

    Christian wrote:
    > Daniel Pitts schrieb:
    >> Arne Vajhøj wrote:
    >>>> Even though I have not tested Derby with a 400 MB database, then I
    >>>> am sure it can handle it.
    >>>
    >>> Translation:
    >>>
    >>> MB = small
    >>> GB = medium
    >>> TB = large
    >>> PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

    >> You mean: PB = Google
    >>

    > nope not Really.. I googled a and found some blog entry from 2006
    > estimating Googles data in compressed form to 220TB. I doubt it grew to
    > a PB since then.
    >
    > http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html
    >

    Compressed form...

    --
    Daniel Pitts' Tech Blog: <http://virtualinfinity.net/wordpress/>
     
    Daniel Pitts, Mar 4, 2008
    #11
  12. Daniel Pitts wrote:
    > Christian wrote:
    >> Daniel Pitts schrieb:
    >>> Arne Vajhøj wrote:
    >>>>> Even though I have not tested Derby with a 400 MB database, then I
    >>>>> am sure it can handle it.
    >>>>
    >>>> Translation:
    >>>>
    >>>> MB = small
    >>>> GB = medium
    >>>> TB = large
    >>>> PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
    >>> You mean: PB = Google
    >>>

    >> nope not Really.. I googled a and found some blog entry from 2006
    >> estimating Googles data in compressed form to 220TB. I doubt it grew
    >> to a PB since then.
    >>
    >> http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html
    >>

    > Compressed form...


    I am a bit surprised that it is no more data.

    According to:
    http://research.google.com/archive/disk_failures.pdf

    Google has >100000 disks 80-400 GB.

    100000*100 GB is 10 PB.

    Arne
     
    Arne Vajhøj, Mar 4, 2008
    #12
  13. Lionel van den Berg

    Arne Vajhøj Guest

    Roedy Green wrote:
    > On 3 Mar 2008 21:19:08 +1100, Lionel van den Berg <>
    > wrote, quoted or indirectly quoted someone who said :
    >> I'm looking into Java DB, and although Sun claims that it can be a full-
    >> scale DBMS I'm wondering what the views on this are? Can Java DB be used
    >> for large databases?

    >
    > Java DB (aka Derby, aka Cloudscape) is for small databases. MySQL
    > which Sun now owns is for intermediate sized ones.


    Los Alamos apparently runs an intermediate sized database:
    http://www.mysql.com/why-mysql/case-studies/mysql-losalamos-casestudy.pdf

    Arne
     
    Arne Vajhøj, Mar 4, 2008
    #13
  14. Lionel van den Berg

    Roedy Green Guest

    On Mon, 03 Mar 2008 21:33:40 -0500, Arne Vajhøj <>
    wrote, quoted or indirectly quoted someone who said :

    >Los Alamos apparently runs an intermediate sized database:
    > http://www.mysql.com/why-mysql/case-studies/mysql-losalamos-casestudy.pdf


    Even some Mickey Mouse databases can manage terabyte plus datasets.
    The catch is what is your transaction volume, and how much data do you
    comb out at a pop.

    The good news is RAM is getting cheaper and disks faster. This means
    databases get more capable by the day even if the software does not
    change.

    Exxon sometimes buys enough RAM to float the entire database into
    cache to get the performance they want.
    --

    Roedy Green Canadian Mind Products
    The Java Glossary
    http://mindprod.com
     
    Roedy Green, Mar 4, 2008
    #14
  15. On Mon, 03 Mar 2008 13:16:12 +0100, Thomas Kellerer wrote:
    > You might want to post that question to the Derby mailing list, support
    > over there is quite good.


    Thanks. Sort of trying to get the unbiased opinion, but it's still worth
    a try :).
     
    Lionel van den Berg, Mar 4, 2008
    #15
  16. On Mon, 03 Mar 2008 21:19:08 +1100, Lionel van den Berg wrote:

    > I'm looking into Java DB, and although Sun claims that it can be a full-
    > scale DBMS I'm wondering what the views on this are? Can Java DB be used
    > for large databases?


    It's easier to say in one post, thanks to all the responses, I have
    enough food for thought :).
     
    Lionel van den Berg, Mar 4, 2008
    #16
  17. Lionel van den Berg

    Guest

    Hi Lionel,

    Java DB (Apache Derby) can definitely handle such a dataset size. We
    have run some performance tests in multi-user mode with databases >
    30+GB.

    You can also check about feedback on using large datasets at:
    http://www.ohloh.net/projects/3816

    Derby is suitable for medium to high-level departmental type of
    databases. It was built originally (back at Cloudscape 10+ years ago)
    with no particular limitation in mind. The engine has always been
    multi-threaded, with a cost-based optimizer and a storage engine that
    can handle large datasets. Of course and as some people have pointed
    here, large datasets can mean different things to a lot of people.

    If you still need to, I would recommend you to post on the derby-user
    mailing list, which is very active by the way:
    http://db.apache.org/derby/derby_comm.html

    You can also browse the list via nabble at
    http://www.nabble.com/Derby-f356.html

    Regards,

    Francois Orsini

    On 3 Mar, 03:47, Lionel van den Berg <> wrote:
    > On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:
    > > Lionel van den Berg, 03.03.2008 11:19:
    > >> I'm looking into Java DB, and although Sun claims that it can be a
    > >> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    > >> be used for large databases?

    >
    > > Define "large"

    >
    > Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
    > do enough to get by.
    >
    > Let's say 10 tables with an average of 4 fields with each table having up
    > to 10 million entries.
    >
    > Not sure if that constitutes large, I'm sure it's not to google.
     
    , Mar 4, 2008
    #17
  18. Thomas Kellerer wrote:
    > Arne Vajhøj wrote on 03.03.2008 22:46:
    >> Lionel van den Berg wrote:
    >>> On Mon, 03 Mar 2008 11:45:37 +0100, Thomas Kellerer wrote:
    >>>> Lionel van den Berg, 03.03.2008 11:19:
    >>>>> I'm looking into Java DB, and although Sun claims that it can be a
    >>>>> full- scale DBMS I'm wondering what the views on this are? Can Java DB
    >>>>> be used for large databases?
    >>>> Define "large"
    >>>
    >>> Sigh, I knew that was coming. I'm not sure, I'm not a database
    >>> person, I do enough to get by.
    >>>
    >>> Let's say 10 tables with an average of 4 fields with each table
    >>> having up to 10 million entries.

    >>
    >> If it is integer fields then 10*4*10M = 400 MB which is a small
    >> database.
    >>

    > The actual data size is not necessarily an indicator whether a database
    > is "large". 10 Million rows (per table) could be a problem, if e.g. the
    > access path to retrieve a subset is not well chosen by the engine and it
    > has to scan all rows in order to retrieve e.g. 5 of them.


    I would phrase it as: large/small depends on size, but whether
    a database is suited for a given problem may depend on many other
    factors than the size of the database.

    Arne
     
    Arne Vajhøj, Mar 17, 2008
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. William Ryan
    Replies:
    0
    Views:
    440
    William Ryan
    Oct 3, 2003
  2. Cowboy \(Gregory A. Beamer\)
    Replies:
    1
    Views:
    1,257
    Guinness Mann
    Oct 6, 2003
  3. =?ISO-8859-1?Q?Andree_Gro=DFe?=

    Re: Java & Databases

    =?ISO-8859-1?Q?Andree_Gro=DFe?=, Jun 30, 2003, in forum: Java
    Replies:
    0
    Views:
    789
    =?ISO-8859-1?Q?Andree_Gro=DFe?=
    Jun 30, 2003
  4. Ken North
    Replies:
    0
    Views:
    618
    Ken North
    Jul 14, 2005
  5. Replies:
    2
    Views:
    1,001
    =?ISO-8859-1?Q?Arne_Vajh=F8j?=
    Oct 14, 2007
Loading...

Share This Page