REXML

[pdg]

Hi All,

As a first exercise with Ruby, I am going through the Pickaxe book and
creating a jukebox. I haven't even tried to create an array of songs
yet, because I got distracted and wanted to work this out. I am trying
toi feed in the data from my iTunes xml file to it to get the data, I
can get it to work if I delete most of the xml file, but when it's 5-6
gig, rexml just seems to die. I have vaguely heard that stream parsing
may be the answer, but am totally unaware of how to use it.

here is the code in my xml reading program so far (saample.rb basically
just creates song items):

require 'rexml/document'
require "sample.rb"

doc = File.open("iTunes.xml")
xml = REXML:ocument.new(doc)
name = "name"
artist = "artist"
time = 60
cnt = 0
xml.elements.each("//key") do |k|
if k.text == "Name" then
name = k.next_sibling.text
cnt += 1
end
if k.text == "Artist" then
artist = k.next_sibling.text
end
if k.text == "Total Time" then
time = k.next_sibling.text.to_i/1000.0
song = Song.new(name,artist,time)
song.to_s

end

end
puts cnt

 

[David Vallner]

--------------enigE2A79B62D516B5259EC7C5CD
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi All,
=20
As a first exercise with Ruby, I am going through the Pickaxe book and
creating a jukebox. I haven't even tried to create an array of songs
yet, because I got distracted and wanted to work this out. I am trying
toi feed in the data from my iTunes xml file to it to get the data, I
can get it to work if I delete most of the xml file, but when it's 5-6
gig,=20

OMFG. That's a -huge- XML file. Probably all of my MP3s together would
fit into there with base64-encoded contents

rexml just seems to die. I have vaguely heard that stream parsing
may be the answer, but am totally unaware of how to use it.
=20

Well, time to learn. I probably never even saw a computer that could
handle a XML file that size using straightforward DOM parsing - which
normally "blows up" the original XML document's size in bytes five times
and more. And REXML definitely doesn't have performance of any kind
amongst its qualities. (And for completeness' sake, I never 'clicked'
with the API either, but I'm a minority there.)

You want a Ruby binding to a stream or pull parser - to my best
knowledge, REXML is neither. That means libxml2, expat, or Xerces.
Compiling Required - I think the one-click installer comes with one of
these, buggered if I know which.

After that, Google is your friend. Look at the documentation to
whichever parser you decided to use and use that - personally, I don't
do much / no non-tree XML parsing at all, so I'm mainly guessing around
on this. The main difference is that while with REXML, you can
arbitrarily look around the XML document, with stream and pull parsing,
you can only process the document in order, and have to keep the state
of that processing (e.g. which track you're currently "working on") in
your Ruby code.

David Vallner


--------------enigE2A79B62D516B5259EC7C5CD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFT78zy6MhrS8astoRAguHAJ97J03rjIz6XEAn4LailVdSfPm1rQCeKMSe
RDd1v9vUj7NWB3NbMWDeJ4w=
=eXTJ
-----END PGP SIGNATURE-----

--------------enigE2A79B62D516B5259EC7C5CD--

 

[Aaron Patterson]

OMFG. That's a -huge- XML file. Probably all of my MP3s together would
fit into there with base64-encoded contents


Well, time to learn. I probably never even saw a computer that could
handle a XML file that size using straightforward DOM parsing - which
normally "blows up" the original XML document's size in bytes five times
and more. And REXML definitely doesn't have performance of any kind
amongst its qualities. (And for completeness' sake, I never 'clicked'
with the API either, but I'm a minority there.)

You want a Ruby binding to a stream or pull parser - to my best
knowledge, REXML is neither. That means libxml2, expat, or Xerces.
Compiling Required - I think the one-click installer comes with one of
these, buggered if I know which.

Ruby comes with a pull parser in the standard lib:
http://ruby-doc.org/stdlib/libdoc/rexml/rdoc/classes/REXML/Parsers/PullParser.html

I would give it a try on a document that large.

 

[Jeff Wood]

OMFG. That's a -huge- XML file. Probably all of my MP3s together would
fit into there with base64-encoded contents



Well, time to learn. I probably never even saw a computer that could
handle a XML file that size using straightforward DOM parsing - which
normally "blows up" the original XML document's size in bytes five times
and more. And REXML definitely doesn't have performance of any kind
amongst its qualities. (And for completeness' sake, I never 'clicked'
with the API either, but I'm a minority there.)

You want a Ruby binding to a stream or pull parser - to my best
knowledge, REXML is neither. That means libxml2, expat, or Xerces.
Compiling Required - I think the one-click installer comes with one of
these, buggered if I know which.

After that, Google is your friend. Look at the documentation to
whichever parser you decided to use and use that - personally, I don't
do much / no non-tree XML parsing at all, so I'm mainly guessing around
on this. The main difference is that while with REXML, you can
arbitrarily look around the XML document, with stream and pull parsing,
you can only process the document in order, and have to keep the state
of that processing (e.g. which track you're currently "working on") in
your Ruby code.

David Vallner

Actually, I recently had to rewrite an xml parser to go stream ( SAX )
style ... REXML made the task VERY easy ...

Yes, it's not the fastest thing there is, but it was "fast enough" ...

Definitely try writing it with REXML before taking the route of anything
heavier.

jd

 

[skt]

I wish I had the foggiest idea of what you guys were talking about.
(Roobist here)
I'm still working on Y's book.

Actually, I recently had to rewrite an xml parser to go stream ( SAX )
style ... REXML made the task VERY easy ...

Yes, it's not the fastest thing there is, but it was "fast enough" ...

Definitely try writing it with REXML before taking the route of anything
heavier.

jd

--
You have a new sung; unsung.
I sing a song falling upon deaf ears,
unsung.

skt
([email protected])
www.freewebs.com/scottygiveshighfives

 

[Chilkat Software]

Is that a mistake? Out of curiosity I took a look on my wife's computer
(she's the iPod user) and her XML file was only 231KB. The structure
of it conforms to the code you shared, so I know it's the right file...

Did you mean to say MB instead of GB?

-Matt

 

[James Edward Gray II]

I probably never even saw a computer that could
handle a XML file that size using straightforward DOM parsing

This is off-topic but I have a theory that it's possible using a
variant of the Flyweight pattern with index offsets into the document
and reparsing individual tags on demand. (I would use weak
referencing to cache them after a parse.)

I've been meaning to code up a proof of concept here and just haven't
had time yet...

You want a Ruby binding to a stream or pull parser - to my best
knowledge, REXML is neither.

REXML includes a stream parser.

James Edward Gray II

 

[David Vallner]

--------------enigDDBC4E770FFD40C2128FFD71
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I wish I had the foggiest idea of what you guys were talking about.
(Roobist here)
I'm still working on Y's book.

Wait... You chimed in on an unrelated thread with a "I don't understand
any of this, FYI" comment?!

The mind, it boggles.

For the record, this isn't a general chat channel. As such, derailing
threads is to be done more subtly

David Vallner


--------------enigDDBC4E770FFD40C2128FFD71
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFT8yIy6MhrS8astoRAu4EAJ45NWr1/Dq7ye7/DzJVZJphPtmbNgCfVe/C
rbQY/SvP3GyjPeS9HX6koc0=
=Ruiz
-----END PGP SIGNATURE-----

--------------enigDDBC4E770FFD40C2128FFD71--

 

[David Vallner]

--------------enig86EA47FF1FDB7F7A8171C2DD
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

REXML includes a stream parser.
=20

So it does, my bad.

David Vallner


--------------enig86EA47FF1FDB7F7A8171C2DD
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFT86Gy6MhrS8astoRAohVAJ0SJ9PA6HGuO/lWMIh9t0OsNbI1RgCeN/Qu
r7kNg+gt02vjHqdeBg71YRI=
=UBXW
-----END PGP SIGNATURE-----

--------------enig86EA47FF1FDB7F7A8171C2DD--

 

[Mark T]

Best to lean towards a database approach when you get to large files.
Neat thing working with XML & REX.
Then you can go to SleepyCat DBxml.
Though the routines are different, that's fer sure.
Someone has a neat Ruby lib for it out there.
Away from my machines for details.

Markt

 

[David Vallner]

--------------enig2297B94195C3E7E26F3CAB4A
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Best to lean towards a database approach when you get to large files.
Neat thing working with XML & REX.
Then you can go to SleepyCat DBxml.
Though the routines are different, that's fer sure.
Someone has a neat Ruby lib for it out there.
Away from my machines for details.
=20
Markt
=20

He's not the one creating the file. So unless you can persuade Apple to
use a XML DB to store iTunes playlists...

(PS: The whole concept of XML DBs is an abomination. The XML Infoset
concept looks like a bloated cloudfest compared to relational data
storage...)

David Vallner


--------------enig2297B94195C3E7E26F3CAB4A
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFT+Y4y6MhrS8astoRAlt/AJ9MHEKFGF8EUI0Z0G4kVec4xhRs1ACbBmQF
ZOF3j0lMbdAm4UmCvIWWCT8=
=y0Mp
-----END PGP SIGNATURE-----

--------------enig2297B94195C3E7E26F3CAB4A--

 

[pdg]

Assuming I go with the Ruby pull parser, how do I use this in my code.
I see from the link the code sample, but I have no idea how to throw
that into my code and make it work. Any suggestions.

Thanks for the discussion so far.

PS: idiot (slaps head). Yes it was 5-6meg not gig!

 

[David Vallner]

--------------enigF9D6B5236ACE2603700BC85A
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Assuming I go with the Ruby pull parser, how do I use this in my code.
I see from the link the code sample, but I have no idea how to throw
that into my code and make it work. Any suggestions.
=20

Generally, you should have some layer between XML input, and processing
the records themselves. E.g. a trivial Song class, or at least a hash.
Personally, I'd make a XMLSongList class that's enumerable (implements
#each), and rework the REXML code that works for small files into one
that yields a Song object for each of the records in succession by
querying the tree accordingly.

That shouldn't then be too hard to rework so that while #each is
running, it opens a pull parser, and for each yield, builds up a Song
object going through the record in the order how the elements appear in
the XML file, instead of a random one. Once you isolate the code that
manipulates the XML to the smallest significant unit (a song record in
this case, I presume), it shouldn't be conceptually that difficult to
rework from a tree parser to a pull parser. The code probably will get a
little messier and verbose, but the main shift of thinking is in not
asking the XML for what your object needs, but feeding an object what
the XML has.

PS: idiot (slaps head). Yes it was 5-6meg not gig!
=20

6MB is still Huge (tm) for a XML file.


--------------enigF9D6B5236ACE2603700BC85A
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)

iD8DBQFFT+5ay6MhrS8astoRAkkFAJ9G4g4kyo6mdKIywnBnXWJ9XpexGACeODK3
GDwxjcj8VJX8nctYiLUgV0I=
=Jh35
-----END PGP SIGNATURE-----

--------------enigF9D6B5236ACE2603700BC85A--

 

[pdg]

Hi David (or others)

I am still not sure I get it, could you explain a little more?

Thanks,
Paul.

 

[Chilkat Software]

I tested the Chilkat XML parser (an in-memory DOM) on a 21MB XML file
that looks like this:

<phonebook>
<address><company>yuy25uiFfaku</company><street>A7ZbA3jP48rp</street><city>fSgWAhn3i3lD</city><state>p3rfNqf6kzUq</state><postal_code>lqVZ0b4daYWQ</postal_code><country>VjfXvb0AdxSt</country><extra>TEST</extra></address>
<address><company>Ki78Ypx8FlbZ</company><street>340PK6u2DsZQ</street><city>EqbFawBo0mCi</city><state>fTZK5YT0Tur8</state><postal_code>EXP29c5Hi2Hj</postal_code><country>sfGB4EzWR3Ft</country><extra>TEST</extra></address>
...
</phonebook>

(the data is random garbage...)

The XML is parsed in 11.5 seconds on a 18.Ghz Pentium 4. Peak memory
usage is 180MB.
I don't think the parser would break a sweat on the 6MB file... ;-)

I uploaded the XML test data to:
http://www.example-code.com/downloads/bigXml.zip
The code for parsing the iTunes XML is easy:
http://www.example-code.com/ruby/ruby-parse-itunes-xml.asp

-Matt

 

[pdg]

Hi thanks for the link, it seems to be working much better, but...

It's getting to about the 1000th file and doing its job, but then
returning the following error:

undefined method 'NextSibling' for nil:NilClass (noMethodError) from
rtunes rb (which is basically just your sample code).

Does this mean I have a broken xml file? Or is something else the
matter?

 

[pdg]

Figured it out.

It was running into problems with videos and podcasts as they don't
have an artist tag.
Got round it by searching for children tags with the title of "Movie"
or "Podcast", and not running the print statements for those items.

Hi thanks for the link, it seems to be working much better, but...

It's getting to about the 1000th file and doing its job, but then
returning the following error:

undefined method 'NextSibling' for nil:NilClass (noMethodError) from
rtunes rb (which is basically just your sample code).

Does this mean I have a broken xml file? Or is something else the
matter?

 

[Chilkat Software]

If you want, send me your test code + file and I'll have a look...

(tomorrow morning though)

Don't worry about a large attachment, just send it zipped...

Best Regards,
Matt

 

[Chilkat Software]

Thanks,

By the way, the Chilkat XML parser is not better than
REXML, it's just different. To give you a little history, it was originally
developed about 7 years ago to handle:

1) Large XML data files where the MSXML parser was s-l-o-w.
At one point, I remember Chilkat XML parsing files in a few seconds
that took MSXML minutes to parse. However, since then MSXML has
improved to the point where it's as good or better in speed...

2) I wanted to create a parser that was forgiving with errors.
Back then, it was a nightmare to have a large XML file with one
small error, perhaps a byte or two that didn't fit the charset encoding,
that would prevent the entire document from loading.

3) I wanted a parser that made it easy to do the common tasks
I'm always faced with in XML -- such as reading/writing config files.

4) I wanted to make it easy to do things not normally handled in
an API -- sorting, compression, encryption, loading / encoding binary
data, etc.

If you give it a try -- let me know what you think. Send me a request for
an example or two and I'll be happy to provide what I can...

-Matt

 

[Glenn Smith]

Is the rexml website down or is it just me?

Thanks,

By the way, the Chilkat XML parser is not better than
REXML, it's just different. To give you a little history, it was originally
developed about 7 years ago to handle:

1) Large XML data files where the MSXML parser was s-l-o-w.
At one point, I remember Chilkat XML parsing files in a few seconds
that took MSXML minutes to parse. However, since then MSXML has
improved to the point where it's as good or better in speed...

2) I wanted to create a parser that was forgiving with errors.
Back then, it was a nightmare to have a large XML file with one
small error, perhaps a byte or two that didn't fit the charset encoding,
that would prevent the entire document from loading.

3) I wanted a parser that made it easy to do the common tasks
I'm always faced with in XML -- such as reading/writing config files.

4) I wanted to make it easy to do things not normally handled in
an API -- sorting, compression, encryption, loading / encoding binary
data, etc.

If you give it a try -- let me know what you think. Send me a request for
an example or two and I'll be happy to provide what I can...

-Matt