Tokenize

Discussion in 'Python' started by Ken Fettig, Jul 24, 2003.

  1. Ken Fettig

    Ken Fettig Guest

    Does Python have an equivelent to the Java StringTokenizer? If so, what is
    it and how do you implement it?
    Thanks
    Ken Fettig

    Ken Fettig, Jul 24, 2003
    #1
    1. Advertising

  2. Ken Fettig

    Alan Kennedy Guest

    Ken Fettig wrote:

    > Does Python have an equivelent to the Java StringTokenizer? If so,
    > what is it and how do you implement it?


    Is this the kind of thing that you mean?

    Python 2.3b1 (#40, Apr 25 2003, 19:06:24)
    Type "help", "copyright", "credits" or "license" for more information.

    >>> s = "This is a string to be tokenised"
    >>> s.split()

    ['This', 'is', 'a', 'string', 'to', 'be', 'tokenised']

    >>> s = "This:is:a:string:to:be:tokenised"
    >>> s.split(':')

    ['This', 'is', 'a', 'string', 'to', 'be', 'tokenised']

    >>> s.split(':', 2)

    ['This', 'is', 'a:string:to:be:tokenised']
    >>>


    Or maybe you have something more specific in mind?

    --
    alan kennedy
    -----------------------------------------------------
    check http headers here: http://xhaus.com/headers
    email alan: http://xhaus.com/mailto/alan
    Alan Kennedy, Jul 24, 2003
    #2
    1. Advertising

  3. Ken Fettig

    Andrew Dalke Guest

    Ken Fetting wants a 'StringTokenizer'.

    Alan Kennedy points out
    > >>> s = "This is a string to be tokenised"
    > >>> s.split()

    > ['This', 'is', 'a', 'string', 'to', 'be', 'tokenised']

    ...
    > Or maybe you have something more specific in mind?


    Another option is the little-known 'shlex' module, part of the standard
    library.

    >>> import shlex, StringIO
    >>> infile = StringIO.StringIO("""ls -lart "have space.*" will travel""")
    >>> x = shlex.shlex(infile)
    >>> x.get_token()

    'ls'
    >>> x.get_token()

    '-'
    >>> x.get_token()

    'lart'
    >>> x.get_token()

    '"have space.*"'
    >>> x.get_token()

    'will'
    >>> x.get_token()

    'travel'
    >>> x.get_token()

    ''
    >>>


    As you can see, it treats '-' unexpectedly (compared to the shell).
    Also, with __iter__ in newer Pythons, if these module were useful
    then it would be nice if "for token in shlex..." worked.

    Andrew
    Andrew Dalke, Jul 24, 2003
    #3
  4. Ken Fettig

    Harry George Guest

    "Ken Fettig" <> writes:

    > Does Python have an equivelent to the Java StringTokenizer? If so, what is
    > it and how do you implement it?
    > Thanks
    > Ken Fettig
    >
    >
    >
    >
    >


    See shlex (in the main distribution) or see a variety of lexer/parser
    tools such as Ply, Yapp.py,

    shlex is about the level of complexity you want. See the Library
    Reference Manual for instructions.

    --

    6-6M31 Knowledge Management
    Phone: (425) 342-5601
    Harry George, Jul 24, 2003
    #4
  5. Andrew Dalke schrieb:

    > Another option is the little-known 'shlex' module, part of the standard
    > library.

    ....
    > As you can see, it treats '-' unexpectedly (compared to the shell).


    This is why shellword was written (see
    <http://www.crazy-compilers.com/py-lib/>)

    Regards
    Hartmut Goebel
    --
    | Hartmut Goebel | We build the crazy compilers |
    | | Compiler Manufacturer |
    Hartmut Goebel, Jul 28, 2003
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lans
    Replies:
    9
    Views:
    470
    Chris \( Val \)
    Jul 10, 2003
  2. Kelvin@!!!

    tokenize a string

    Kelvin@!!!, Feb 24, 2005, in forum: C++
    Replies:
    4
    Views:
    7,886
  3. Ram Laxman

    How can I use strtok for tokenize integers

    Ram Laxman, Apr 11, 2004, in forum: C Programming
    Replies:
    6
    Views:
    1,305
    Christopher Benson-Manica
    Apr 12, 2004
  4. Bjorn Pettersen

    RE: Tokenize

    Bjorn Pettersen, Jul 24, 2003, in forum: Python
    Replies:
    0
    Views:
    395
    Bjorn Pettersen
    Jul 24, 2003
  5. Andr? Roberge
    Replies:
    1
    Views:
    500
    Tim Peters
    Sep 10, 2004
Loading...

Share This Page