Easy way to remove HTML entities from an HTML document?

R

Robert Oschler

Is there a module/function to remove all the HTML entities from an HTML
document (e.g. - &nbsp, &amp, &apos, etc.)?

If not I'll just write one myself but I figured I'd save myself some time.

Thanks,
 
C

Christopher T King

Is there a module/function to remove all the HTML entities from an HTML
document (e.g. - &nbsp, &amp, &apos, etc.)?
Click to expand...

htmllib has this capability, but if you're not doing any other HTML
parsing, a regex, coupled with htmllib's helper module, htmlentitydefs,
does nicely:

import re
import htmlentitydefs

def convertentity(m):
if m.group(1)=='#':
try:
return chr(int(m.group(2)))
except ValueError:
return '&#%s;' % m.group(2)
try:
return htmlentitydefs.entitydefs[m.group(2)]
except KeyError:
return '&%s;' % m.group(2)

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convert,s)

converthtml('Some &lt;html&gt; string.') # --> 'Some <html> string.'

Unknown or invalid entities are left in &xxx; format, while also leaving
Unicode entities in format. If you want a Unicode string to be
returned (and Unicode entities interpreted), replace 'chr' with 'unichr',
and 'htmlentitydefs.entitydefs[m.group(2)]' with
'unichr(htmlentitydefs.name2codepoint[m.group(2)])'.

Hope this helps.
 
R

Robert Oschler

Christopher T King said:
htmllib has this capability, but if you're not doing any other HTML
parsing, a regex, coupled with htmllib's helper module, htmlentitydefs,
does nicely:

import re
import htmlentitydefs

def convertentity(m):
if m.group(1)=='#':
try:
return chr(int(m.group(2)))
except ValueError:
return '&#%s;' % m.group(2)
try:
return htmlentitydefs.entitydefs[m.group(2)]
except KeyError:
return '&%s;' % m.group(2)

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convert,s)

converthtml('Some &lt;html&gt; string.') # --> 'Some <html> string.'

Unknown or invalid entities are left in &xxx; format, while also leaving
Unicode entities in format. If you want a Unicode string to be
returned (and Unicode entities interpreted), replace 'chr' with 'unichr',
and 'htmlentitydefs.entitydefs[m.group(2)]' with
'unichr(htmlentitydefs.name2codepoint[m.group(2)])'.

Hope this helps.
Click to expand...

Chris,

I believe the line that reads:

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convert,s)

Should read:

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convertentity,s)

Once I made that change it worked like a charm. I'm showing the correction
for future Usenet searchers.

So you can pass a function to re.sub() as the replacement patttern? Very
cool, I didn't know that. I think you could spend a year just learning
regular expressions and still miss something.


Thanks,
Robert.
 
C

Christopher T King

I believe the line that reads:

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convert,s)

Should read:

def converthtml(s):
return re.sub(r'&(#?)(.+?);',convertentity,s)
Click to expand...

Oops, you're right, mea culpa :)
So you can pass a function to re.sub() as the replacement patttern? Very
cool, I didn't know that. I think you could spend a year just learning
regular expressions and still miss something.
Click to expand...

That feature is only mentioned briefly in the online docs, and not at all
in sre.sub's docstring. Surprising, since it's indeed a very useful
feature.
 
R

Robert Oschler

Christopher T King said:
That feature is only mentioned briefly in the online docs, and not at all
in sre.sub's docstring. Surprising, since it's indeed a very useful
feature.
Click to expand...

Chris,

Speaking of learning cool things by osmosis, do you know of a well commented
source of Python code, perhaps an Open Source project, that I could study to
learn more interesting techniques like the regexp tip you shared? I find
that studying other people's code is the best way to avoid getting in a
programming rut.

Thanks.
 
C

Christopher T King

Speaking of learning cool things by osmosis, do you know of a well commented
source of Python code, perhaps an Open Source project, that I could study to
learn more interesting techniques like the regexp tip you shared? I find
that studying other people's code is the best way to avoid getting in a
programming rut.
Click to expand...

I seem to recall reading about that re.sub trick in something linked from
Pythonware's Daily Python URL (http://www.pythonware.com/daily/). There
are often links there to interesting and useful code snippets from
ActiveState's Python Cookbook and other sources; I'd say start there if
you want to find neat tricks you can do with Python.

I'm not sure of any particularly "well commented" Python projects though
(I've never really looked into that), but you'll probably find some
interesting small projects in the Vaults of Parnassus
(http://www.vex.net/parnassus/).
 
R

Robert Oschler

Christopher T King said:
I seem to recall reading about that re.sub trick in something linked from
Pythonware's Daily Python URL (http://www.pythonware.com/daily/). There
are often links there to interesting and useful code snippets from
ActiveState's Python Cookbook and other sources; I'd say start there if
you want to find neat tricks you can do with Python.

I'm not sure of any particularly "well commented" Python projects though
(I've never really looked into that), but you'll probably find some
interesting small projects in the Vaults of Parnassus
(http://www.vex.net/parnassus/).
Click to expand...

Thanks Chris and thanks for all your other help.

With your Python skill you should work for Google. Too bad you don't, you'd
be a wealthy man soon (Google IPO). Wish I did. :)
 
C

Christopher T King

With your Python skill you should work for Google. Too bad you don't,
you'd be a wealthy man soon (Google IPO). Wish I did. :)
Click to expand...

Thanks for the compliment. :) To work at Google is my dream job, and I'm
sure that of many others on this list, too (makes me wonder if any Google
employees read this list...).
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

javax.xml.transform.Transformer and HTML entities 4
Need assistance finetuning HTML, CSS, Javascript - sticky header issue 3
What's the best way to parse this HTML tag? 3
Peasy: an easy but powerful parser 0
.NET-ey way to convert XML-encoded/escaped entities into normal characters/HTML? 2
Ruby, Unicode, and HTML Entities Problem 4
Getting an html file from a online html document and converting itback to code. 7
IDE seems to mangle html 4.0 entities 1

Members online

Total: 34 (members: 1, guests: 33)
Robots: 204

Forum statistics

Threads
473,766
Messages
2,569,569
Members
45,042
Latest member
icassiem

Latest Threads

Top