Howto or Tutorial for tokenize module for a newbie?

[TY]

Hi,

Can someone point me to a Howto or Tutorial for tokenize module for a
newbie? I read the documentation but it doesn't have a lot of info...
Thanks!

 

[sreekant]

Hi,

Can someone point me to a Howto or Tutorial for tokenize module for a
newbie? I read the documentation but it doesn't have a lot of info...
Thanks!

Hi there

I don't know if I got your requirement. But I used a own version of
tokenizer which can use more than one delimiters. It is below. Hopefully
of some use to you.




def gettokens(dat):
delims={'=':'',';':',','=':'','(':'',')':'',':':'','[':'',']':''}
wlist=[]
appended=0
for n in string.split(dat,'\n'):
word=''
for m in n:
appended=0
if delims.has_key(m):
wlist.append(word)
wlist.append(m)
word=''
appended=1
else:
word=word+m
if appended==0:
wlist.append(word)
appended=1
wlist.append("\n")
return wlist

I am sure there are plenty of ways to write it better than that.

Good luck
sree

 

[John J. Lee]

Can someone point me to a Howto or Tutorial for tokenize module for a
newbie? I read the documentation but it doesn't have a lot of info...

Here's a simple example:

http://groups.google.com/group/comp.lang.python/msg/e84e36ad10d78303


John

 

[Fuzzyman]

Hi,

Can someone point me to a Howto or Tutorial for tokenize module for a
newbie? I read the documentation but it doesn't have a lot of info...

Here's another useful example :

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52298

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml