Which one to use: generate_tokens or tokenize?

Discussion in 'Python' started by Andr? Roberge, Sep 10, 2004.

  1. According to the Python documentation:

    18.5 tokenize -- Tokenizer for Python source
    ....
    The primary entry point is a generator:
    generate_tokens(readline)
    ....
    An older entry point is retained for backward compatibility:
    tokenize(readline[, tokeneater])
    ====
    Does this mean that one should preferably use generate_tokens? If so,
    what are the advantages?

    André Roberge
     
    Andr? Roberge, Sep 10, 2004
    #1
    1. Advertisements

  2. Andr? Roberge

    Tim Peters Guest

    [Andr? Roberge]
    > According to the Python documentation:
    >
    > 18.5 tokenize -- Tokenizer for Python source
    > ...
    > The primary entry point is a generator:
    > generate_tokens(readline)
    > ...
    > An older entry point is retained for backward compatibility:
    > tokenize(readline[, tokeneater])
    > ====
    > Does this mean that one should preferably use generate_tokens?


    Yes.

    > If so, what are the advantages?


    Be adventurous: try them both. You'll figure it out quickly. If you
    have to endure "an explanation" first, read PEP 255, where
    tokenize.tokenize was used as an example motivating the desirability
    of introducing generators.
     
    Tim Peters, Sep 10, 2004
    #2
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lans
    Replies:
    9
    Views:
    652
    Chris \( Val \)
    Jul 10, 2003
  2. Kelvin@!!!

    tokenize a string

    Kelvin@!!!, Feb 24, 2005, in forum: C++
    Replies:
    4
    Views:
    8,068
  3. Ram Laxman

    How can I use strtok for tokenize integers

    Ram Laxman, Apr 11, 2004, in forum: C Programming
    Replies:
    6
    Views:
    1,601
    Christopher Benson-Manica
    Apr 12, 2004
  4. Ken Fettig

    Tokenize

    Ken Fettig, Jul 24, 2003, in forum: Python
    Replies:
    4
    Views:
    9,499
    Hartmut Goebel
    Jul 28, 2003
  5. qwweeeit
    Replies:
    16
    Views:
    635
    M.E.Farmer
    Apr 22, 2005
  6. glomde
    Replies:
    3
    Views:
    403
    glomde
    May 9, 2006
  7. Lorenzo Thurman

    NEWBIE: Tokenize command output

    Lorenzo Thurman, May 11, 2006, in forum: Python
    Replies:
    8
    Views:
    748
    bruno at modulix
    May 12, 2006
  8. Replies:
    0
    Views:
    380
Loading...