Does StreamTokenizer.pushBack correctly update TT_EOF ?

Discussion in 'Java' started by enokacorea@gmail.com, Sep 19, 2008.

  1. Guest

    Hello there !

    Does StreamTokenizer.pushBack correctly update TT_EOF ?

    I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
    Within the loop I am using sis.nextToken() and sis.pushBack()
    Now, if the eof is reached and pushedBack, will the loop continue one
    more time ?

    If pushBack() happens properly, I think the loop should continue one
    more time.

    Would appreciate any help (I could not get any feedback on this by
    browsing the newsgroups).

    Many thanks in advance.

    Chris
    , Sep 19, 2008
    #1
    1. Advertising

  2. In article
    <>,
    wrote:

    [...]
    > I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
    > Within the loop I am using sis.nextToken() and sis.pushBack()
    > Now, if the eof is reached and pushedBack,


    The "next call to the nextToken method" will again "return the current
    value in the ttype field," TT_EOF.

    <http://java.sun.com/javase/6/docs/api/java/io/StreamTokenizer.html#pushBack()>

    > will the loop continue one more time?


    It shouldn't.

    > If pushBack() happens properly, I think the loop should continue one
    > more time.


    It might be a useful feature, but it's not the documented behavior.

    [...]
    --
    John B. Matthews
    trashgod at gmail dot com
    home dot woh dot rr dot com slash jbmatthews
    John B. Matthews, Sep 19, 2008
    #2
    1. Advertising

  3. Guest

    On Sep 19, 4:53 pm, "John B. Matthews" <> wrote:
    > In article
    > <>,
    >
    >   wrote:
    >
    > [...]
    >
    > > I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
    > > Within the loop I am using sis.nextToken() and sis.pushBack()
    > > Now, if the eof is reached and pushedBack,

    >
    > The "next call to the nextToken method" will again "return the current
    > value in the ttype field," TT_EOF.
    >
    > <http://java.sun.com/javase/6/docs/api/java/io/StreamTokenizer.html#pu...()>
    >
    > > will the loop continue one more time?

    >
    > It shouldn't.
    >
    > > If pushBack() happens properly, I think the loop should continue one
    > > more time.

    >
    > It might be a useful feature, but it's not the documented behavior.
    >
    > [...]
    > --
    > John B. Matthews
    > trashgod at gmail dot com
    > home dot woh dot rr dot com slash jbmatthews


    Many thanks for your feedback John ! Much appreciated.

    I had to put in an additional OR condition in the loop to get the
    right behaviour

    while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {

    .... where notyeteof is set in the body of the loop as follows :...

    notyeteof=false;
    if (sis.ttype!=sis.TT_EOF) {
    sis.nextToken();
    if (sis.ttype==sis.TT_EOF) {notyeteof=true;}
    .... more statements .........
    }

    }
    , Sep 19, 2008
    #3
  4. In article
    <>,
    wrote:

    > On Sep 19, 4:53 pm, "John B. Matthews" <> wrote:
    > > In article
    > > <>,
    > >   wrote:
    > > [...]
    > > > I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
    > > > Within the loop I am using sis.nextToken() and sis.pushBack()
    > > > Now, if the eof is reached and pushedBack,

    > >
    > > The "next call to the nextToken method" will again "return the current
    > > value in the ttype field," TT_EOF.
    > >
    > > <http://java.sun.com/javase/6/docs/api/java/io/StreamTokenizer.html#pu...()>
    > >
    > > > will the loop continue one more time?

    > >
    > > It shouldn't.
    > >
    > > > If pushBack() happens properly, I think the loop should continue one
    > > > more time.

    > >
    > > It might be a useful feature, but it's not the documented behavior.
    > >
    > > [...]

    >
    > Many thanks for your feedback John ! Much appreciated.
    >
    > I had to put in an additional OR condition in the loop to get the
    > right behaviour
    >
    > while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {

    [...]

    Excellent! Assuming correct initialization, the predicate may be
    simplified to:

    while (sis.ttype != StreamTokenizer.TT_EOF || notYetEOF) {

    I have to say this still seems counterintuitive. What's left to do after
    seeing TT_EOF?

    --
    John B. Matthews
    trashgod at gmail dot com
    home dot woh dot rr dot com slash jbmatthews
    John B. Matthews, Sep 19, 2008
    #4
  5. Guest

    On Sep 19, 11:22 pm, "John B. Matthews" <> wrote:
    > In article
    > <>,
    >
    >
    >
    >
    >
    >   wrote:
    > > On Sep 19, 4:53 pm, "John B. Matthews" <> wrote:
    > > > In article
    > > > <>,
    > > >   wrote:
    > > > [...]
    > > > > I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
    > > > > Within the loop I am using sis.nextToken() and sis.pushBack()
    > > > > Now, if the eof is reached and pushedBack,

    >
    > > > The "next call to the nextToken method" will again "return the current
    > > > value in the ttype field," TT_EOF.

    >
    > > > <http://java.sun.com/javase/6/docs/api/java/io/StreamTokenizer.html#pu...()>

    >
    > > > > will the loop continue one more time?

    >
    > > > It shouldn't.

    >
    > > > > If pushBack() happens properly, I think the loop should continue one
    > > > > more time.

    >
    > > > It might be a useful feature, but it's not the documented behavior.

    >
    > > > [...]

    >
    > > Many thanks for your feedback John ! Much appreciated.

    >
    > > I had to put in an additional OR condition in the loop to get the
    > > right behaviour

    >
    > > while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {

    >
    > [...]
    >
    > Excellent! Assuming correct initialization, the predicate may be
    > simplified to:
    >
    > while (sis.ttype != StreamTokenizer.TT_EOF || notYetEOF) {
    >
    > I have to say this still seems counterintuitive. What's left to do after
    > seeing TT_EOF?
    >
    > --
    > John B. Matthews
    > trashgod at gmail dot com
    > home dot woh dot rr dot com slash jbmatthews- Hide quoted text -
    >
    > - Show quoted text -


    You see, if I push back something near the end of a loop I want it
    processed at the start of the next loop. If TT_EOF is not set
    correctly the last token will not be processed correctly.

    This is what I tried to resolve with the logic I described earlier.

    I am still having my original problem though !

    Trying to debug this I tried a small test and found that - nextToken()
    immediately followed by pushBack() does not give the same result as
    not having both statements !!

    Any thoughts on this ?
    , Sep 20, 2008
    #5
  6. In article
    <>,
    wrote:
    [...]
    > > I have to say this still seems counterintuitive. What's left to do after
    > > seeing TT_EOF?

    > [...]
    > You see, if I push back something near the end of a loop I want it
    > processed at the start of the next loop. If TT_EOF is not set
    > correctly the last token will not be processed correctly.
    >
    > This is what I tried to resolve with the logic I described earlier.
    >
    > I am still having my original problem though !
    >
    > Trying to debug this I tried a small test and found that - nextToken()
    > immediately followed by pushBack() does not give the same result as
    > not having both statements !!
    >
    > Any thoughts on this ?


    Yes, the effect of pushBack() is limited to giving the same result from
    nextToken(); nval and sval are unchanged. The mechanism seems intended
    to provide inexpensive, one-token look-ahead, rather than unlimited undo.

    <code>
    import java.io.*;
    public class Tokenizer {
    public static void main(String[] args) throws IOException {
    StreamTokenizer tokens = new
    StreamTokenizer(new StringReader("Test"));
    int token;
    token = tokens.nextToken();
    print(tokens, token);
    token = tokens.nextToken();
    print(tokens, token);
    tokens.pushBack();
    token = tokens.nextToken();
    print(tokens, token);
    }
    private static void print(StreamTokenizer tokens, int token) {
    System.out.println((token == StreamTokenizer.TT_EOF)
    + " " + tokens.sval);
    }
    }
    </code>
    <console>
    false Test
    true null
    true null
    </console>

    Here's a more elaborate example:

    <http://sites.google.com/site/drjohnbmatthews/enumerated-functions>

    Notice how encountering TT_EOF (defined as Symbol.END) signals
    successful parsing; any other result represents a (not otherwise
    identified) syntax error.

    --
    John B. Matthews
    trashgod at gmail dot com
    home dot woh dot rr dot com slash jbmatthews
    John B. Matthews, Sep 20, 2008
    #6
  7. Guest

    On Sep 20, 7:44 pm, "John B. Matthews" <> wrote:
    > In article
    > <>,  wrote:
    >
    > [...]
    >
    >
    >
    >
    >
    > > > I have to say this still seems counterintuitive. What's left to do after
    > > > seeing TT_EOF?

    > > [...]
    > > You see, if I push back something near the end of a loop I want it
    > > processed at the start of the next loop. If TT_EOF is not set
    > > correctly the last token will not be processed correctly.

    >
    > > This is what I tried to resolve with the logic I described earlier.

    >
    > > I am still having my original problem though !

    >
    > > Trying to debug this I tried a small test and found that - nextToken()
    > > immediately followed by pushBack() does not give the same result as
    > > not having both statements !!

    >
    > > Any thoughts on this ?

    >
    > Yes, the effect of pushBack() is limited to giving the same result from
    > nextToken(); nval and sval are unchanged. The mechanism seems intended
    > to provide inexpensive, one-token look-ahead, rather than unlimited undo.
    >
    > <code>
    > import java.io.*;
    > public class Tokenizer {
    >     public static void main(String[] args) throws IOException {
    >         StreamTokenizer tokens = new
    >             StreamTokenizer(new StringReader("Test"));
    >         int token;
    >         token = tokens.nextToken();
    >         print(tokens, token);
    >         token = tokens.nextToken();
    >         print(tokens, token);
    >         tokens.pushBack();
    >         token = tokens.nextToken();
    >         print(tokens, token);
    >     }
    >     private static void print(StreamTokenizer tokens, int token) {
    >         System.out.println((token == StreamTokenizer.TT_EOF)
    >             + " "  + tokens.sval);
    >     }}
    >
    > </code>
    > <console>
    > false Test
    > true null
    > true null
    > </console>
    >
    > Here's a more elaborate example:
    >
    > <http://sites.google.com/site/drjohnbmatthews/enumerated-functions>
    >
    > Notice how encountering TT_EOF (defined as Symbol.END) signals
    > successful parsing; any other result represents a (not otherwise
    > identified) syntax error.
    >
    > --
    > John B. Matthews
    > trashgod at gmail dot com
    > home dot woh dot rr dot com slash jbmatthews- Hide quoted text -
    >
    > - Show quoted text -


    Many thanks once again for the code snippet and reference.
    The snippet of course works as expected.

    My issue is that I thought my code should give the identical output
    either if
    a) I insert a readToken() immediately followed by a pushBack() or
    b) with neither of these statements
    (all else being the same).

    I find that this is not the case.

    After further debugging I removed

    while (sis.ttype != StreamTokenizer.TT_EOF) and put in a

    while (!EOF)

    where EOF is set if (sis.ttype == StreamTokenizer.TT_EOF)
    immediately after the "chief" nextToken() - which is not subject to
    pushBack().

    This solved the problem !!

    By the way, I visited your web site and was intrigued to find that you
    are a doctor of medicine in addition to being a computer expert !

    Chris
    , Sep 21, 2008
    #7
  8. In article
    <>,
    wrote:
    [...]
    > My issue is that I thought my code should give the identical output
    > either if a) I insert a readToken() immediately followed by a
    > pushBack() or b) with neither of these statements (all else being the
    > same). I find that this is not the case.


    I am unable to reproduce what you describe:

    <code>
    import java.io.*;
    public class Tokenizer {
    public static void main(String[] args) throws IOException {
    StreamTokenizer tokens = new
    StreamTokenizer(new StringReader("Test 123"));
    int token;
    token = tokens.nextToken();
    print(tokens, token);

    token = tokens.nextToken();
    print(tokens, token);

    tokens.pushBack();
    tokens.pushBack();
    tokens.pushBack();
    token = tokens.nextToken();
    tokens.pushBack();
    token = tokens.nextToken();
    tokens.pushBack();
    token = tokens.nextToken();
    print(tokens, token);

    token = tokens.nextToken();
    print(tokens, token);
    }
    private static void print(StreamTokenizer tokens, int token) {
    System.out.println((token == StreamTokenizer.TT_EOF)
    + " " + tokens.sval + " " + tokens.nval);
    }
    }
    </code>
    <console>
    false Test 0.0
    false null 123.0
    false null 123.0
    true null 123.0
    </console>

    > After further debugging I removed
    >
    > while (sis.ttype != StreamTokenizer.TT_EOF) and put in a
    >
    > while (!EOF)
    >
    > where EOF is set if (sis.ttype == StreamTokenizer.TT_EOF) immediately
    > after the "chief" nextToken() - which is not subject to pushBack().
    >
    > This solved the problem !!


    Excellent! I would also develop tests to ensure that your code rejects
    invalid input.

    > By the way, I visited your web site and was intrigued to find that you
    > are a doctor of medicine in addition to being a computer expert !


    I'm no expert, but sometimes I apply software engineering principles to
    medical informatics. I rarely appear on TV. :)

    --
    John B. Matthews
    trashgod at gmail dot com
    home dot woh dot rr dot com slash jbmatthews
    John B. Matthews, Sep 21, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. D. Lane
    Replies:
    1
    Views:
    2,480
    Dave Rathnow
    Jun 30, 2003
  2. John C. Bollinger

    Re: How does StreamTokenizer work

    John C. Bollinger, Jun 30, 2003, in forum: Java
    Replies:
    0
    Views:
    855
    John C. Bollinger
    Jun 30, 2003
  3. Chandrashekar

    Pushback

    Chandrashekar, Jul 14, 2003, in forum: C++
    Replies:
    4
    Views:
    583
    Victor Bazarov
    Jul 14, 2003
  4. Lordy
    Replies:
    1
    Views:
    371
    Dale King
    Jun 11, 2006
  5. john isaac

    vector pushback compilation error

    john isaac, Mar 9, 2006, in forum: C++
    Replies:
    4
    Views:
    345
    Chris Theis
    Mar 9, 2006
Loading...

Share This Page