StreamTokenizer and unicode

Discussion in 'Java' started by ppcguy, Jun 26, 2006.

  1. ppcguy

    ppcguy Guest

    i'm trying to parse a file using StreamTokenizer class.
    the file has strings like "Hello\u1234There".

    StreamTokenizer returns this as a string "Hellou1234There".

    i understand this class is not unicode capable - but any hints
    as to how this could be parsed properly.

    i thought any 'u' followed by 4 hex digits would work because
    i've got some strings where 'u' + hext digits is part of a string.
     
    ppcguy, Jun 26, 2006
    #1
    1. Advertising

  2. ppcguy wrote:
    > i'm trying to parse a file using StreamTokenizer class.
    > the file has strings like "Hello\u1234There".
    >
    > StreamTokenizer returns this as a string "Hellou1234There".
    >
    > i understand this class is not unicode capable - but any hints
    > as to how this could be parsed properly.
    >
    > i thought any 'u' followed by 4 hex digits would work because
    > i've got some strings where 'u' + hext digits is part of a string.


    The u + hex digits is part of Java source code processing, including
    processing of string literals.

    Since you are reading from a Stream, how about applying a filter to do
    any preprocessing you want?

    Patricia
     
    Patricia Shanahan, Jun 26, 2006
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. D. Lane
    Replies:
    1
    Views:
    2,498
    Dave Rathnow
    Jun 30, 2003
  2. John C. Bollinger

    Re: How does StreamTokenizer work

    John C. Bollinger, Jun 30, 2003, in forum: Java
    Replies:
    0
    Views:
    875
    John C. Bollinger
    Jun 30, 2003
  3. Sudsy

    Re: StreamTokenizer

    Sudsy, Jul 24, 2003, in forum: Java
    Replies:
    2
    Views:
    521
    Sudsy
    Aug 2, 2003
  4. Ian deSouza

    Re: StreamTokenizer

    Ian deSouza, Aug 1, 2003, in forum: Java
    Replies:
    0
    Views:
    1,571
    Ian deSouza
    Aug 1, 2003
  5. JVarsoke
    Replies:
    1
    Views:
    1,543
    Dale King
    Oct 1, 2003
Loading...

Share This Page