Huge XML data needed

[Beda Christoph Hammerschmidt]

I wat to perform some performance measurements on an XML database. FOr
this reason i need some huge XML sample data. The data should be not
too structured and a lot of reasonable queries should make sense.
Any idea, where i can get this data ??

 

[Andy Dingley]

Any idea, where i can get this data ??

Make it yourself. That way you can control the size and the
distribution of certain features. If this process is automated, then
you can easily run tests over and over with different parameters.

It's often useful (but rarely done) to test, not just that "it works",
but to test for sensitivity to different sorts of load. Does
performance change with many small items, or with few large items ?
Does sorted/unsorted input data make a difference ?

Another source of "real world" data in a large corporate is to connect
to something like an LDAP server and use that. I've also done much of
my own testing with lists of endangered species form the WCMC. You may
also find the W3C site useful, particularly the RDF test cases (not
large, but they do demonstrate many obscure conditions).

 

[Arto Viitanen]

Beda> I wat to perform some performance measurements on an XML database. FOr
Beda> this reason i need some huge XML sample data. The data should be not
Beda> too structured and a lot of reasonable queries should make sense. Any
Beda> idea, where i can get this data ??

You might get some RSS feed. RSS is a form used by several news servers to
distribute news. So by definition, there is not much structure, but you can
make reasonable queries, like what happed (some terrorist act), what was the
score (some soccer game) etc.

 

[Toivo Lainevool]

I wat to perform some performance measurements on an XML database. FOr
this reason i need some huge XML sample data. The data should be not
too structured and a lot of reasonable queries should make sense.
Any idea, where i can get this data ??

I'm not sure what you mean by "huge", but there is a good amount of
data that might be intersting to query at:
http://www.ibiblio.org/xml/examples/shakespeare/

Toivo Lainevool
http://www.XMLPatterns.com - Develop effective DTDs and XML Schema
documents for your XML using structural design patterns.

 

[Fabien R]

Why don't you generate them ?
Use a free-db like MySQL...

 

[Johannes Koch]

Beda> I wat to perform some performance measurements on an XML database. FOr
Beda> this reason i need some huge XML sample data. The data should be not
Beda> too structured and a lot of reasonable queries should make sense. Any
Beda> idea, where i can get this data ??

You might get some RSS feed.

But RSS - by definition - is not "huge XML data".

 

[Arto Viitanen]

Beda> this reason i need some huge XML sample data. The data should be not
Beda> too structured and a lot of reasonable queries should make sense. Any
Beda> idea, where i can get this data ??
Johannes> But RSS - by definition - is not "huge XML data".

But I got two out of third: it is not too structured and there can be
reasonable queries !

 

[Johannes Koch]

Beda> this reason i need some huge XML sample data. The data should be not
Beda> too structured and a lot of reasonable queries should make sense. Any
Beda> idea, where i can get this data ??

Johannes> But RSS - by definition - is not "huge XML data".

But I got two out of third: it is not too structured and there can be
reasonable queries !

That's right

 

[Akmal B. Chaudhri]

Why don't you generate them ?

Good idea. There are 5 major XML DB Benchmark efforts. Some include data
generators. See:

http://www.rpbourret.com/xml/XMLDBLinks.htm#Benchmarks

Ron Bourret has a link to a benchmark page that I use to maintain, but I
no longer have time to maintain it.

Use a free-db like MySQL...

Some benchmarks and performance issues are also covered in the book I
helped edit:

A.B. Chaudhri, A. Rashid and R. Zicari (eds.) (2003) XML data management:
native XML and XML-enabled database systems (Reading, Massachusetts:
Addison-Wesley)

http://www.awprofessional.com/titles/0201844524/

HTH

akmal

 

[Stefan Ram]

I wat to perform some performance measurements on an XML database. FOr
this reason i need some huge XML sample data. The data should be not
too structured and a lot of reasonable queries should make sense.

This is somewhat structured, but large:

http://rdf.dmoz.org/rdf/content.rdf.u8.gz

 

[Andy Dingley]

But RSS - by definition - is not "huge XML data".

Much RSS isn't even XML !

Today's RSS feed bug was this
http://www.littlefluffy.com/index.php?a=rss

<description>
A more aptly named game you are not likely to find. [...] a great
game for drug users &lt;em>and&lt;/em> kids.
</description>

 

[Patrick TJ McPhee]

% Today's RSS feed bug was this
% http://www.littlefluffy.com/index.php?a=rss
%
% <description>
% A more aptly named game you are not likely to find. [...] a great
% game for drug users &lt;em>and&lt;/em> kids.
% </description>

So, what's wrong with it? That <em> should appear as mark-up, or that
you think > shouldn't be there?

 

[Andy Dingley]

% game for drug users &lt;em>and&lt;/em> kids.

So, what's wrong with it? That <em> should appear as mark-up, or that
you think > shouldn't be there?

There's no valid encoding for HTML in RSS where the opening character
of a tag is escaped, but not the closing character.

 

[Patrick TJ McPhee]

% On Sun, 4 Apr 2004 03:00:54 +0200 (MEST), (e-mail address removed) (Patrick
% TJ McPhee) wrote:
%
% >% game for drug users &lt;em>and&lt;/em> kids.
%
% >So, what's wrong with it? That <em> should appear as mark-up, or that
% >you think > shouldn't be there?
%
% There's no valid encoding for HTML in RSS where the opening character
% of a tag is escaped, but not the closing character.

OK, perhaps it's not valid RSS, but it's valid XML.

 

[Jimmy Zhang]

http://www.pir.uniprot.org/database/download.shtml

up to a gig in size

 

[Torsten Bittner]

too structured and a lot of reasonable queries should make sense.

Any idea, where i can get this data ??

I also suggest you to create the data on your own. I recommend ToXgene:
http://www.cs.toronto.edu/tox/toxgene

hth
Torsten