Tokenize

[Ken Fettig]

Does Python have an equivelent to the Java StringTokenizer? If so, what is
it and how do you implement it?
Thanks
Ken Fettig
(e-mail address removed)
(e-mail address removed)

 

[Alan Kennedy]

Does Python have an equivelent to the Java StringTokenizer? If so,
what is it and how do you implement it?

Is this the kind of thing that you mean?

Python 2.3b1 (#40, Apr 25 2003, 19:06:24)
Type "help", "copyright", "credits" or "license" for more information.
['This', 'is', 'a', 'string', 'to', 'be', 'tokenised']
['This', 'is', 'a', 'string', 'to', 'be', 'tokenised']

s.split(':', 2) ['This', 'is', 'a:string:to:be:tokenised']

Or maybe you have something more specific in mind?

 

[Andrew Dalke]

Ken Fetting wants a 'StringTokenizer'.

Alan Kennedy points out

['This', 'is', 'a', 'string', 'to', 'be', 'tokenised'] ...
Or maybe you have something more specific in mind?

Another option is the little-known 'shlex' module, part of the standard
library.

As you can see, it treats '-' unexpectedly (compared to the shell).
Also, with __iter__ in newer Pythons, if these module were useful
then it would be nice if "for token in shlex..." worked.

Andrew
(e-mail address removed)

 

[Harry George]

Does Python have an equivelent to the Java StringTokenizer? If so, what is
it and how do you implement it?
Thanks
Ken Fettig
(e-mail address removed)
(e-mail address removed)

See shlex (in the main distribution) or see a variety of lexer/parser
tools such as Ply, Yapp.py,

shlex is about the level of complexity you want. See the Library
Reference Manual for instructions.

 

[Hartmut Goebel]

Another option is the little-known 'shlex' module, part of the standard
library. ....
As you can see, it treats '-' unexpectedly (compared to the shell).

This is why shellword was written (see
<http://www.crazy-compilers.com/py-lib/>)

Regards
Hartmut Goebel