Splitting with Regular Expressions

[qwweeeit]

Splitting with RE has (for me!) misterious behaviour!

I want to get the words from this string:
s= 'This+(that)= a.string!!!'

in a list like that ['This', 'that', 'a.string']
considering "a.string" as a word.

Python 2.3.4 (#2, Aug 19 2004, 15:49:40)
[GCC 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

s= 'This+(that)= a.string!!!'
import re
p=re.compile('[\W].')
p.split(s)

['This', 'that', '', '', 'tring', '!']

p=re.compile('[\W.]')
p.split(s)

['This', '', 'that', '', '', 'a', 'string', '', '', '']

Help!

 

[Thomas Guettler]

Am Thu, 17 Mar 2005 06:51:19 -0800 schrieb qwweeeit:

Splitting with RE has (for me!) misterious behaviour!

I want to get the words from this string:
s= 'This+(that)= a.string!!!'

in a list like that ['This', 'that', 'a.string']
considering "a.string" as a word.

Hi,

try this:

re.findall(r'[\w\.]+', s)
['This', 'that', 'a.string']

If you use r'...' you don't need to
use \\ if you mean \

\w matches a-zA-Z0-9_
\W matches all except \w

Thomas

 

[Fredrik Lundh]

Splitting with RE has (for me!) misterious behaviour!

I want to get the words from this string:
s= 'This+(that)= a.string!!!'

in a list like that ['This', 'that', 'a.string']
considering "a.string" as a word.

print re.findall("[\w.]+", s)

</F>

 

[Paul McGuire]

A pyparsing example may be less mysterious. You can define words to be
any group of alphas, or you can define a word to be alphas concatenated
by '.'s. scanString is a generator that scans for matches in the input
string and returns the matching token list, and the start and end
location of the match within the input string. (Because it returns a
list, this is why we have to peel of element 0 of each match to get the
word.) See the sample code attached.

-- Paul
(get pyparsing at http://pyparsing.sourceforge.net)


from pyparsing import Word,alphas,delimitedList

test= 'This+(that)= a.string!!! This... is .just.a sentence.'

word = Word(alphas)

print [ wd[0] for wd,s,e in word.scanString(test) ]

# prints ['This', 'that', 'a', 'string', 'This', 'is', 'just', 'a',
'sentence']

word = delimitedList(Word(alphas), delim=".",combine=True)
print [ wd[0] for wd,s,e in word.scanString(test) ]

# prints ['This', 'that', 'a.string', 'This', 'is', 'just.a',
'sentence']

 

[qwweeeit]

I thank you for your help.
The more flexible solution (Paul McGuire) is interesting but i don't
need such a flexibility. In fact I am implementing a cross-reference
tool and working on python sources, I don't need the '.' as separator
in order to capture variables and commands.
I thank nevertheless Paul for the advice on using pyparsing.

The other two solutions (Thomas Guettler and Fredrik Lundh) are quite
similar except for the use of "raw". I can use both.

 

[Fredrik Lundh]

In fact I am implementing a cross-reference tool and working on
python sources, I don't need the '.' as separator in order to capture
variables and commands.

if you're parsing Python source code, consider using the tokenize module:

http://docs.python.org/lib/module-tokenize.html

if you don't have to support "broken" source code, the parse module may
be even more useful:

http://docs.python.org/lib/module-parser.html

for simpler cases, the "class browser" parser may be an even better choice:

http://docs.python.org/lib/module-pyclbr.html

</F>

 

[qwweeeit]

It' s my faute that I have not read more deeply the Library
Reference...
In any case the time "wasted" in developping small applications to
number lines and remove comments, triple quoted strings, multiline
instructions etc. has been "useful" to learn the language...
Now I already have the single tokens of a source saved in a file with
the reference to the corresponding line number.
An example is:

052 PROGNAME
052 sys.argv
052 0
053 AUTHOR
053 .encode
054 VERSION
056 URL_BASE
057 OUTPUT_HTML
058 OUTPUT_RSS
060 CSS
071 urllib.URLopener.version
072 urllib.FancyURLopener.prompt_user_passwd
072 lambda
072 self
072 host
072 realm
072 None
072 None
074 categories

The corresponding source is
(from http://inigo.katxi.org/devel/misc/googlenews.py):

052 PROGNAME = sys.argv[0]
053 AUTHOR = u'Iñigo Serna'.encode('utf-8')
054 VERSION = '0.3'
055
056 URL_BASE = 'http://news.google.com'
057 OUTPUT_HTML = 'news-%s-%s.html'
058 OUTPUT_RSS = 'news-%s-%s.xml'
059
060 CSS = """<style type="text/css">
061 body { color: black; background: white; }
062 a { color : #003399; text-decoration : none; }
063 a:hover { color : #339900; text-decoration : none; }
064 a.main { font-size : 100%; }
065 span.text { font-size : 100%; }
066 span.data { color : #666666; font-size : 80%; }
067 a.other { color : #003366; font-size : 75%; }
068 a.other:hover { color : #339900; font-size : 75%; }
069 </style>"""
070
071 urllib.URLopener.version = 'Mozilla/4.0 (compatible; MSIE 5.5;
Windows NT 5.0; T312461)'
072 urllib.FancyURLopener.prompt_user_passwd = lambda self, host,
realm: (None, None)
073
074 categories = ['w', 'n', 'b', 't', 's', 'e', 'm']

As you can see there are no literal strings nor comments (they are
saved in an accompagning file).
Now I have "only" to sort them, display in a table or (if you prefer)
in an extended display in which all the references to lines are
expanded...
Of course many tokens (like for, in, if, not, etc.) will be eliminated
for space reasons and also because I already know how they are used.

I applied some years ago the same approach to understand an assembler
source...

In the case of python (and all others high level languages) it should
be very useful also the function tree or flow chart...