Which one to use: generate_tokens or tokenize?

Discussion in 'Python' started by Andr? Roberge, Sep 10, 2004.

  1. According to the Python documentation:

    18.5 tokenize -- Tokenizer for Python source
    ....
    The primary entry point is a generator:
    generate_tokens(readline)
    ....
    An older entry point is retained for backward compatibility:
    tokenize(readline[, tokeneater])
    ====
    Does this mean that one should preferably use generate_tokens? If so,
    what are the advantages?

    André Roberge
    Andr? Roberge, Sep 10, 2004
    #1
    1. Advertising

  2. Andr? Roberge

    Tim Peters Guest

    [Andr? Roberge]
    > According to the Python documentation:
    >
    > 18.5 tokenize -- Tokenizer for Python source
    > ...
    > The primary entry point is a generator:
    > generate_tokens(readline)
    > ...
    > An older entry point is retained for backward compatibility:
    > tokenize(readline[, tokeneater])
    > ====
    > Does this mean that one should preferably use generate_tokens?


    Yes.

    > If so, what are the advantages?


    Be adventurous: try them both. You'll figure it out quickly. If you
    have to endure "an explanation" first, read PEP 255, where
    tokenize.tokenize was used as an example motivating the desirability
    of introducing generators.
    Tim Peters, Sep 10, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Lans
    Replies:
    9
    Views:
    470
    Chris \( Val \)
    Jul 10, 2003
  2. Kelvin@!!!

    tokenize a string

    Kelvin@!!!, Feb 24, 2005, in forum: C++
    Replies:
    4
    Views:
    7,886
  3. Ram Laxman

    How can I use strtok for tokenize integers

    Ram Laxman, Apr 11, 2004, in forum: C Programming
    Replies:
    6
    Views:
    1,305
    Christopher Benson-Manica
    Apr 12, 2004
  4. glomde
    Replies:
    3
    Views:
    288
    glomde
    May 9, 2006
  5. Replies:
    0
    Views:
    316
Loading...

Share This Page