Stripping unwanted html

[Wild Al]

Hi everyone:

I'm trying to strip html with the exception of a few html tags.

I have found the following code:

def strip_tags(html)
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)

while token = tokenizer.next
node = HTML::Node.parse(nil, 0, 0, token, false)
# result is only the content of any Text nodes
text << node.to_s if node.class == HTML::Text
end
# strip any comments, and if they have a newline at the end (ie.
line with
# only a comment) strip that too
text.gsub(/<!--(.*?)-->[\n]?/m, "")
else
html # already plain text
end
end

I'm trying to understand what is going on in this code but cannot find
documenation for HTML::Tokenizer or HTML::Node.parse. Does anyone know
the use of the parameters in the parse method?

In the while loop, how do you access the html tag. If I could access
the html tags, I could then decide if I wanted to keep the tag or not.

Thanks for reading,
Wild Al

 

[Michael Moen]

Hi everyone:

I'm trying to strip html with the exception of a few html tags.

I have found the following code:

def strip_tags(html)
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)

while token = tokenizer.next
node = HTML::Node.parse(nil, 0, 0, token, false)
# result is only the content of any Text nodes
text << node.to_s if node.class == HTML::Text
end
# strip any comments, and if they have a newline at the end (ie.
line with
# only a comment) strip that too
text.gsub(/<!--(.*?)-->[\n]?/m, "")
else
html # already plain text
end
end

I'm trying to understand what is going on in this code but cannot find
documenation for HTML::Tokenizer or HTML::Node.parse. Does anyone know
the use of the parameters in the parse method?

In the while loop, how do you access the html tag. If I could access
the html tags, I could then decide if I wanted to keep the tag or not.

Thanks for reading,
Wild Al

Al- I recently needed a parser similar to Perl's HTML::Scrubber. <a
href="http://www.underpantsgnome.com/2006/09/09/using-hpricot-to-scrub-html/">This</a>
is what I came up with , you may find it useful.

Michael

 

[eden]

Hey, that's RoR's strip_tags method

I ran into the same issue so went down your path and instead of
fretting about no docs, just hacked my way through using IRB.

Here's a modified version of what I came up with, maybe you'll find it
useful?

def strip_tags_except(html, exceptions = [])
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)
while token = tokenizer.next
case node = HTML::Node.parse(nil, 0, 0, token, false)
when HTML::Tag
text << node.to_s if exceptions.include?(node.name)
when HTML::Text
text << node.to_s
end
end
text
else
html
end
end

The one I had also stripped attributes and closed up dangling tags if
it found any. Have a look at RoR's strip_links for more examples of
HTML::Node/HTML::Tokenizer usage.

 

[eden]

Have a look at RoR's strip_links for more examples of
HTML::Node/HTML::Tokenizer usage.

Sorry, I meant sanitize:
http://api.rubyonrails.com/classes/ActionView/Helpers/TextHelper.html#M000516

 

[Wild Al]

Here's a modified version of what I came up with, maybe you'll find it
useful?

def strip_tags_except(html, exceptions = [])
if html.index("<")
text = ""
tokenizer = HTML::Tokenizer.new(html)
while token = tokenizer.next
case node = HTML::Node.parse(nil, 0, 0, token, false)
when HTML::Tag
text << node.to_s if exceptions.include?(node.name)
when HTML::Text
text << node.to_s
end
end
text
else
html
end
end

I found this method very useful; it is exactly what I needed. Thanks.
To all others: your suggestions helped too, especially in understanding
ruby. Thanks again...

 

[eden li]

Glad I could help. One security-related caveat.

The method I posted doesn't strip attributes, so it may be possible for
someone to "hack" your site by putting javascript onto one of the
allowed tags.

You can fix that by changing the last line of the first branch of the
if statement from "text" to "sanitize(text)", eg:

if html.index("<")
...
sanitize(text)
else
...