"Soup Strainer" for ElementSoup?

[erikcw]

Hi all,

I was reading in the Beautiful Soup documentation that you should use
a "Soup Strainer" object to keep memory usage down.

Since I'm already using Element Tree elsewhere in the project, I
figured it would make sense to use ElementSoup to keep the api
consistent. (and cElementTree should be faster right??).

I can't seem to figure out how to pass ElementSoup a "soup strainer"
though.

Any ideas?

Also - do I need to use the extract() method with ElementSoup like I
do with Beautiful Soup to keep garbage collection working?

Thanks!
Erik

 

[John Nagle]

Hi all,

I was reading in the Beautiful Soup documentation that you should use
a "Soup Strainer" object to keep memory usage down.

Since I'm already using Element Tree elsewhere in the project, I
figured it would make sense to use ElementSoup to keep the api
consistent. (and cElementTree should be faster right??).

I can't seem to figure out how to pass ElementSoup a "soup strainer"
though.

Any ideas?

Also - do I need to use the extract() method with ElementSoup like I
do with Beautiful Soup to keep garbage collection working?

Thanks!
Erik

I really should get my version of BeautifulSoup merged back into
the mainstream. I have one that's been modified to use weak pointers
for all "up" and "left" links, which makes the graph cycle free. So
the memory is recovered by reference count update as soon as you
let go of the head of the tree. That helps with the garbage problem.

What are you parsing? If you're parsing well-formed XML,
BeautifulSoup is overkill. If you're parsing real-world HTML,
ElementTree is too brittle.

John Nagle

 

[erikcw]

erikcwwrote:







I really should get my version of BeautifulSoup merged back into
the mainstream. I have one that's been modified to use weak pointers
for all "up" and "left" links, which makes the graph cycle free. So
the memory is recovered by reference count update as soon as you
let go of the head of the tree. That helps with the garbage problem.

What are you parsing? If you're parsing well-formed XML,
BeautifulSoup is overkill. If you're parsing real-world HTML,
ElementTree is too brittle.

John Nagle

I'm parsing real-world HTML with BeautifulSoup and XML with
cElementTree.

I'm guessing that the only benefit to using ElementSoup is that I'll
have one less API to keep track of, right? Or are there memory
benefits in converting the Soup object to an ElementTree?

Any idea about using a Soup Strainer with ElementSoup?

Thanks!

 

[Stefan Behnel]

I'm parsing real-world HTML with BeautifulSoup and XML with
cElementTree.

I'm guessing that the only benefit to using ElementSoup is that I'll
have one less API to keep track of, right?

If your "real-world" HTML is still somewhat close to HTML, lxml.html might be
an option. It combines the ElementTree API with a good close-to-HTML parser
and some helpful HTML handling tools.

http://codespeak.net/lxml
http://codespeak.net/lxml/lxmlhtml.html

You can also use it with the BeautifulSoup parser if you really need to.

http://codespeak.net/lxml/elementsoup.html

Stefan

 

[Fredrik Lundh]

I'm parsing real-world HTML with BeautifulSoup and XML with
cElementTree.

I'm guessing that the only benefit to using ElementSoup is that I'll
have one less API to keep track of, right? Or are there memory
benefits in converting the Soup object to an ElementTree?

It's purely an API thing: ElementSoup loads the entire HTML file with
BeautifulSoup, and then uses the resulting BS data structure to build an
ET tree.

The ET tree doesn't contain cycles, though, so you can safely pull out
the strings you need from ET and throw away the rest of the tree.

Any idea about using a Soup Strainer with ElementSoup?

The strainer is used when parsing the file, to control what goes into
the BS tree; to add straining support to ES, you could e.g. add a
parseOnlyThese option that's passed through to BS.

</F>