Java DB, is it for small databases only?

[Lionel van den Berg]

I'm looking into Java DB, and although Sun claims that it can be a full-
scale DBMS I'm wondering what the views on this are? Can Java DB be used
for large databases?

Thanks

Lionel.

 

[Thomas Kellerer]

Lionel van den Berg, 03.03.2008 11:19:

I'm looking into Java DB, and although Sun claims that it can be a full-
scale DBMS I'm wondering what the views on this are? Can Java DB be used
for large databases?

Define "large"

 

[Lionel van den Berg]

Lionel van den Berg, 03.03.2008 11:19:

Define "large"

Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
do enough to get by.

Let's say 10 tables with an average of 4 fields with each table having up
to 10 million entries.

Not sure if that constitutes large, I'm sure it's not to google.

 

[Thomas Kellerer]

Lionel van den Berg, 03.03.2008 12:47:

Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
do enough to get by.

Let's say 10 tables with an average of 4 fields with each table having up
to 10 million entries.

Not sure if that constitutes large, I'm sure it's not to google.

OK, I would think that qualifies as "large" (at least not as small )

I have no experience with Derby using such a setup. I would be a bit hesitating about using it for such a database. But I guess this can only be answered by setting up a testdatabase of that size.

You might want to post that question to the Derby mailing list, support over there is quite good.

Thomas

 

[Arne Vajhøj]

Sigh, I knew that was coming. I'm not sure, I'm not a database person, I
do enough to get by.

Let's say 10 tables with an average of 4 fields with each table having up
to 10 million entries.

If it is integer fields then 10*4*10M = 400 MB which is a small
database.

Even though I have not tested Derby with a 400 MB database, then I
am sure it can handle it.

Arne

 

[Arne Vajhøj]

If it is integer fields then 10*4*10M = 400 MB which is a small
database.

Even though I have not tested Derby with a 400 MB database, then I
am sure it can handle it.

Translation:

MB = small
GB = medium
TB = large
PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

Arne

 

[Daniel Pitts]

Translation:

MB = small
GB = medium
TB = large
PB = arghhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

You mean: PB = Google

 

[Thomas Kellerer]

Arne Vajhøj wrote on 03.03.2008 22:46:

If it is integer fields then 10*4*10M = 400 MB which is a small
database.

The actual data size is not necessarily an indicator whether a database is
"large". 10 Million rows (per table) could be a problem, if e.g. the access path
to retrieve a subset is not well chosen by the engine and it has to scan all
rows in order to retrieve e.g. 5 of them.

If the number of updates and parallel reads is very high, but the design does
not allow for concurrent updates and reads ("writers blocking readers") then the
performance of the application will suffer as.
Btw: writers blocking readers is a problem even for some of the "big" names

Thomas

 

[Roedy Green]

I'm looking into Java DB, and although Sun claims that it can be a full-
scale DBMS I'm wondering what the views on this are? Can Java DB be used
for large databases?

Java DB (aka Derby, aka Cloudscape) is for small databases. MySQL
which Sun now owns is for intermediate sized ones.

For alternatives running the gamut, see
http://mindprod.com/jgloss/sqlvendors.html

 

[Christian]

You mean: PB = Google

nope not Really.. I googled a and found some blog entry from 2006
estimating Googles data in compressed form to 220TB. I doubt it grew to
a PB since then.

http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html

 

[Daniel Pitts]

nope not Really.. I googled a and found some blog entry from 2006
estimating Googles data in compressed form to 220TB. I doubt it grew to
a PB since then.

http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html

Compressed form...

 

[Arne Vajhøj]

Compressed form...

I am a bit surprised that it is no more data.

According to:
http://research.google.com/archive/disk_failures.pdf

Google has >100000 disks 80-400 GB.

100000*100 GB is 10 PB.

Arne

 

[Arne Vajhøj]

Java DB (aka Derby, aka Cloudscape) is for small databases. MySQL
which Sun now owns is for intermediate sized ones.

Los Alamos apparently runs an intermediate sized database:
http://www.mysql.com/why-mysql/case-studies/mysql-losalamos-casestudy.pdf

Arne

 

[Roedy Green]

Los Alamos apparently runs an intermediate sized database:
http://www.mysql.com/why-mysql/case-studies/mysql-losalamos-casestudy.pdf

Even some Mickey Mouse databases can manage terabyte plus datasets.
The catch is what is your transaction volume, and how much data do you
comb out at a pop.

The good news is RAM is getting cheaper and disks faster. This means
databases get more capable by the day even if the software does not
change.

Exxon sometimes buys enough RAM to float the entire database into
cache to get the performance they want.

 

[Lionel van den Berg]

You might want to post that question to the Derby mailing list, support
over there is quite good.

Thanks. Sort of trying to get the unbiased opinion, but it's still worth
a try .

 

[Lionel van den Berg]

I'm looking into Java DB, and although Sun claims that it can be a full-
scale DBMS I'm wondering what the views on this are? Can Java DB be used
for large databases?

It's easier to say in one post, thanks to all the responses, I have
enough food for thought .

 

[francois.orsini]

Hi Lionel,

Java DB (Apache Derby) can definitely handle such a dataset size. We
have run some performance tests in multi-user mode with databases >
30+GB.

You can also check about feedback on using large datasets at:
http://www.ohloh.net/projects/3816

Derby is suitable for medium to high-level departmental type of
databases. It was built originally (back at Cloudscape 10+ years ago)
with no particular limitation in mind. The engine has always been
multi-threaded, with a cost-based optimizer and a storage engine that
can handle large datasets. Of course and as some people have pointed
here, large datasets can mean different things to a lot of people.

If you still need to, I would recommend you to post on the derby-user
mailing list, which is very active by the way:
http://db.apache.org/derby/derby_comm.html

You can also browse the list via nabble at
http://www.nabble.com/Derby-f356.html

Regards,

Francois Orsini

 

[Arne Vajhøj]

Arne Vajhøj wrote on 03.03.2008 22:46:
The actual data size is not necessarily an indicator whether a database
is "large". 10 Million rows (per table) could be a problem, if e.g. the
access path to retrieve a subset is not well chosen by the engine and it
has to scan all rows in order to retrieve e.g. 5 of them.

I would phrase it as: large/small depends on size, but whether
a database is suited for a given problem may depend on many other
factors than the size of the database.

Arne