RE Module

[Roman]

I am trying to filter a column in a list of all html tags.

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The results I get are sporatic. Sometimes two tags are removed.
Sometimes 1 tag is removed. Sometimes no tags are removed. Could
somebody tell me where have I gone wrong here?

Thanks in advance

 

[Simon Forman]

I am trying to filter a column in a list of all html tags.
What?

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The results I get are sporatic. Sometimes two tags are removed.
Sometimes 1 tag is removed. Sometimes no tags are removed. Could
somebody tell me where have I gone wrong here?

Thanks in advance

I'm no re expert, so I won't try to advise you on your re, but it might
help those who are if you gave examples of your input and output data.
What results are you getting for what input strings.

Also, if you're just trying to strip html markup to get plain text from
a file, "w3m -dump some.html" works great. ;-)

HTH,
~Simon

 

[Anthra Norell]

Roman,

Your re works for me. I suspect you have tags spanning lines, a thing you get more often than not. If so, processing linewise
doesn't work. You need to catch the tags like this:

If your text is reasonably small I would recommend this solution. Else you might want to take a look at SE which is a stream edtor
that does the buffering for you:

http://cheeseshop.python.org/pypi/SE/2.2 beta
(... your text without tags ...)

The Tag_Stripper is made up of two regexes. The second one catches comments which may nest tags. The first expression alone would
also catch comments, but would mistake the '>' of the first nested tag for the end of the comment and quit prematurely. The example
"re.sub ('<(.|\n)*?>', '', text)" above would misperform in this respect.

Your Tag_Stripper takes input from files directly:
'name_of_output_file'

Or if you want to to view the output:
(... your text without tags ...)

If you want to keep the definitions for later use, do this:

Tag_Stripper.save ('[your_path/]tag_stripper.se')

Your definitions are now saved in the file 'tag_stripper.se'. You can edit that file. The next time you need a Tag_Stripper you can
make it simply by naming the file:

Tag_Stripper = SE.SE ('[your_path/]tag_stripper.se')

You can easily expand the capabilities of your Tag_Stripper. If, for instance, you want to translate the ampersand escapes (&nbsp;
etc.) you'd simply add the name of the file that defines the ampersand replacements:

'htm2iso.se' comes with the SE package ready to use and as an example for writing ones own replacement sets.


Frederic


----- Original Message -----
From: "Simon Forman" <[email protected]>
Newsgroups: comp.lang.python
To: <[email protected]>
Sent: Friday, August 25, 2006 7:09 AM
Subject: Re: RE Module

I am trying to filter a column in a list of all html tags.
What?

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The results I get are sporatic. Sometimes two tags are removed.
Sometimes 1 tag is removed. Sometimes no tags are removed. Could
somebody tell me where have I gone wrong here?

Thanks in advance

I'm no re expert, so I won't try to advise you on your re, but it might
help those who are if you gave examples of your input and output data.
What results are you getting for what input strings.

Also, if you're just trying to strip html markup to get plain text from
a file, "w3m -dump some.html" works great. ;-)

HTH,
~Simon

 

[Roman]

Thanks for your help.

A thing I didn't mention is that before the statement row[0] =
re.sub(r'<.*?>', '', row[0]), I have row[0]=re.sub(r'[^
0-9A-Za-z\"\'\.\,\#\@\!\(\)\*\&\%\%\\\/\:\;\?\`\~\<\>]', '', row[0])
statement. Hence, the line separators are going to be gone. You
mentioned the size of the string could be a factor. If so what is the
max size before I see problems?

Thanks again

Roman,

Your re works for me. I suspect you have tags spanning lines, a thing you get more often than not. If so, processing linewise
doesn't work. You need to catch the tags like this:

If your text is reasonably small I would recommend this solution. Else you might want to take a look at SE which is a stream edtor
that does the buffering for you:

http://cheeseshop.python.org/pypi/SE/2.2 beta
(... your text without tags ...)

The Tag_Stripper is made up of two regexes. The second one catches comments which may nest tags. The first expression alone would
also catch comments, but would mistake the '>' of the first nested tag for the end of the comment and quit prematurely. The example
"re.sub ('<(.|\n)*?>', '', text)" above would misperform in this respect.

Your Tag_Stripper takes input from files directly:
'name_of_output_file'

Or if you want to to view the output:
(... your text without tags ...)

If you want to keep the definitions for later use, do this:

Tag_Stripper.save ('[your_path/]tag_stripper.se')

Your definitions are now saved in the file 'tag_stripper.se'. You can edit that file. The next time you need a Tag_Stripper you can
make it simply by naming the file:

Tag_Stripper = SE.SE ('[your_path/]tag_stripper.se')

You can easily expand the capabilities of your Tag_Stripper. If, for instance, you want to translate the ampersand escapes (&nbsp;
etc.) you'd simply add the name of the file that defines the ampersand replacements:

'htm2iso.se' comes with the SE package ready to use and as an example for writing ones own replacement sets.


Frederic


----- Original Message -----
From: "Simon Forman" <[email protected]>
Newsgroups: comp.lang.python
To: <[email protected]>
Sent: Friday, August 25, 2006 7:09 AM
Subject: Re: RE Module

I am trying to filter a column in a list of all html tags.
What?

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The results I get are sporatic. Sometimes two tags are removed.
Sometimes 1 tag is removed. Sometimes no tags are removed. Could
somebody tell me where have I gone wrong here?

Thanks in advance

I'm no re expert, so I won't try to advise you on your re, but it might
help those who are if you gave examples of your input and output data.
What results are you getting for what input strings.

Also, if you're just trying to strip html markup to get plain text from
a file, "w3m -dump some.html" works great. ;-)

HTH,
~Simon

 

[tobiah]

I am trying to filter a column in a list of all html tags.

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The regex will be 'greedy' and match through one tag
all the way to the end of another on the same line.
There are more complete suggestions offered, but
it seems to me that the simple fix here is to not
match through the end of the tag, like this:

"<[^>]*>"

 

[Roman]

This is excellent. Thanks a lot.

Also, what made the expression greedy?

I am trying to filter a column in a list of all html tags.

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The regex will be 'greedy' and match through one tag
all the way to the end of another on the same line.
There are more complete suggestions offered, but
it seems to me that the simple fix here is to not
match through the end of the tag, like this:

"<[^>]*>"

 

[Anthra Norell]

Roman,

I don't quite understand what you mean. Line separators gone? That would be the '\n', right? What of it if you process line by line,
as your variable name 'row' suggests?
As to the maximum size re can handle, I have no idea. I vaguely remember the topic being discussed. You should be able to find
the discussions in the archives, if a knowlegeable soul doesn't volunteer the info right away. With SE it is of no concern.

Anyway, I think the best thing to do is to just try with a real page:
( ... page without tags, but lots of empty lines ...)

If you want to take the empty lines out, do this:

"|" means do the preceding replacements (which happen to be deletions: replace with nothing) and go on from there. The expressions
we added say: delete lines that contain only spaces. Do that (another "|"). And finally replace multiple consecutive line feeds with
a single line feed.
So you can develop interactively. Add a definition. See what it does. Add another one. One little step at a time. Hacking at
its best!

Frederic


----- Original Message -----
From: "Roman" <[email protected]>
Newsgroups: comp.lang.python
To: <[email protected]>
Sent: Friday, August 25, 2006 6:14 PM
Subject: Re: RE Module

Thanks for your help.

A thing I didn't mention is that before the statement row[0] =
re.sub(r'<.*?>', '', row[0]), I have row[0]=re.sub(r'[^
0-9A-Za-z\"\'\.\,\#\@\!\(\)\*\&\%\%\\\/\:\;\?\`\~\<\>]', '', row[0])
statement. Hence, the line separators are going to be gone. You
mentioned the size of the string could be a factor. If so what is the
max size before I see problems?

Thanks again

Roman,

Your re works for me. I suspect you have tags spanning lines, a thing you get more often than not. If so, processing linewise
doesn't work. You need to catch the tags like this:

text = re.sub ('<(.|\n)*?>', '', text)

If your text is reasonably small I would recommend this solution. Else you might want to take a look at SE which is a stream edtor
that does the buffering for you:

http://cheeseshop.python.org/pypi/SE/2.2 beta

import SE
Tag_Stripper = SE.SE (' "~<(.|\n)*?>~=" "~<!--(.|\n)*?-->~=" ')
print Tag_Stripper (text)

(... your text without tags ...)

The Tag_Stripper is made up of two regexes. The second one catches comments which may nest tags. The first expression alone would
also catch comments, but would mistake the '>' of the first nested tag for the end of the comment and quit prematurely. The example
"re.sub ('<(.|\n)*?>', '', text)" above would misperform in this respect.

Your Tag_Stripper takes input from files directly:

Tag_Stripper ('name_of_file.htm', 'name_of_output_file')

'name_of_output_file'

Or if you want to to view the output:

Tag_Stripper ('name_of_file.htm', '')

(... your text without tags ...)

If you want to keep the definitions for later use, do this:

Tag_Stripper.save ('[your_path/]tag_stripper.se')

Your definitions are now saved in the file 'tag_stripper.se'. You can edit that file. The next time you need a Tag_Stripper you can
make it simply by naming the file:

Tag_Stripper = SE.SE ('[your_path/]tag_stripper.se')

You can easily expand the capabilities of your Tag_Stripper. If, for instance, you want to translate the ampersand escapes (&nbsp;
etc.) you'd simply add the name of the file that defines the ampersand replacements:

Tag_Stripper = SE.SE ('tag_stripper.se htm2iso.se')

'htm2iso.se' comes with the SE package ready to use and as an example for writing ones own replacement sets.


Frederic


----- Original Message -----
From: "Simon Forman" <[email protected]>
Newsgroups: comp.lang.python
To: <[email protected]>
Sent: Friday, August 25, 2006 7:09 AM
Subject: Re: RE Module

Roman wrote:
I am trying to filter a column in a list of all html tags.

What?

To do that, I have setup the following statement.

row[0] = re.sub(r'<.*?>', '', row[0])

The results I get are sporatic. Sometimes two tags are removed.
Sometimes 1 tag is removed. Sometimes no tags are removed. Could
somebody tell me where have I gone wrong here?

Thanks in advance

I'm no re expert, so I won't try to advise you on your re, but it might
help those who are if you gave examples of your input and output data.
What results are you getting for what input strings.

Also, if you're just trying to strip html markup to get plain text from
a file, "w3m -dump some.html" works great. ;-)

HTH,
~Simon

 

[tobiah]

This is excellent. Thanks a lot.

Also, what made the expression greedy?

They usually are, by default. It means that when there
are more than one ways to match the pattern, choose the
one that matches the most text. Often there are flags
available to change that behavior. I'm not sure off hand
how to do it with the re module.

 

[Tim Chase]

Also, what made the expression greedy?

They usually are, by default. It means that when there
are more than one ways to match the pattern, choose the
one that matches the most text. Often there are flags
available to change that behavior. I'm not sure off hand
how to do it with the re module.

In python's RE module, they're like Perl:

Greedy: "<.*>"
Nongreedy: "<.*?>"

By appending a question-mark onto the operator, one makes it a
non-greedy repeat. It also applies to the plus ("one or more")
and the questionmark ("zero or one")

-tkc

 

[tobiah]

In python's RE module, they're like Perl:

Greedy: "<.*>"
Nongreedy: "<.*?>"

Oh, I have never seen that. In that case, why
did Roman's first example not work well for
HTML tags?

'<.*?>'

Also, how does the engine decide whether I am adjusting
the greed of the previous operator, or just asking
for another possible character?

Suppose I want:

"x*?" to match "xxxxxxxO"

If the '?' means non greedy, then I should get 'x' back.
If the '?' means optional character then I should get
the full string back.

Checking in python:

######################################
import re

s = 'xxxxxxx0'

m = re.search("x*", s)
print "First way", m.group(0)

m = re.search("x*?", s)
print "Second way", m.group(0)
#####################################
First way xxxxxxx
Second way

So now I'm really confused. It didn't do a non-greedy
'x' match, nor did it allow the '?' to match the '0'.

 

[Fredrik Lundh]

Also, how does the engine decide whether I am adjusting
the greed of the previous operator, or just asking
for another possible character?

"?" always modifies the *preceeding* RE element.

if the preceeding element is a pattern (e.g. a character or group), it
means that the pattern is optional.

if the preceeding element is a repeat modifier (*, +, or ?), it changes
the greediness.

Suppose I want:

"x*?" to match "xxxxxxxO"

If the '?' means non greedy, then I should get 'x' back.

no, because "*" means *ZERO* or more matches, not one or more.

If the '?' means optional character then I should get
the full string back.

no, because "?" never means anything on its own; it's a pattern
modifier, not a pattern.

Checking in python:

######################################
import re

s = 'xxxxxxx0'

m = re.search("x*", s)
print "First way", m.group(0)

m = re.search("x*?", s)
print "Second way", m.group(0)
#####################################
First way xxxxxxx
Second way

So now I'm really confused. It didn't do a non-greedy
'x' match, nor did it allow the '?' to match the '0'.

see above. reading the RE documentation again may also help.

</F>

 

[Tim Chase]

######################################

import re

s = 'xxxxxxx0'

m = re.search("x*", s)
print "First way", m.group(0)

m = re.search("x*?", s)
print "Second way", m.group(0)
#####################################
First way xxxxxxx
Second way

So now I'm really confused. It didn't do a non-greedy
'x' match, nor did it allow the '?' to match the '0'.

it did do a non-greedy match. It found as few "x"s as possible.
it found 0 of them, and quit. For a better test, use

s = '<tag 1><tag 2>'
print re.search('<tag.*?>',s).group(0)
print re.search('<tag.*>',s).group(0)

(the question/problem at hand)

-tkc

 

[Roman]

I looked at a book called beginning python and it claims that <.*?> is
a non-greedy match.

 

[tobiah]

I looked at a book called beginning python and it claims that <.*?> is
a non-greedy match.

Yeah, I get that now, but why didn't it work for you in
the first place?

 

[Roman]

It turns out false alarm. It work. I had other logic in the
expression involving punctuation marks and got all confused with the
escape characters. It becomes a mess trying to keep track of all the
reserved character as you are going from module to module.