Does StreamTokenizer.pushBack correctly update TT_EOF ?

E

enokacorea

Hello there !

Does StreamTokenizer.pushBack correctly update TT_EOF ?

I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
Within the loop I am using sis.nextToken() and sis.pushBack()
Now, if the eof is reached and pushedBack, will the loop continue one
more time ?

If pushBack() happens properly, I think the loop should continue one
more time.

Would appreciate any help (I could not get any feedback on this by
browsing the newsgroups).

Many thanks in advance.

Chris
 
J

John B. Matthews

(e-mail address removed) wrote:

[...]
I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
Within the loop I am using sis.nextToken() and sis.pushBack()
Now, if the eof is reached and pushedBack,

The "next call to the nextToken method" will again "return the current
value in the ttype field," TT_EOF.

will the loop continue one more time?

It shouldn't.
If pushBack() happens properly, I think the loop should continue one
more time.

It might be a useful feature, but it's not the documented behavior.

[...]
 
E

enokacorea

 (e-mail address removed) wrote:

[...]
I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
Within the loop I am using sis.nextToken() and sis.pushBack()
Now, if the eof is reached and pushedBack,

The "next call to the nextToken method" will again "return the current
value in the ttype field," TT_EOF.

will the loop continue one more time?

It shouldn't.
If pushBack() happens properly, I think the loop should continue one
more time.

It might be a useful feature, but it's not the documented behavior.

[...]

Many thanks for your feedback John ! Much appreciated.

I had to put in an additional OR condition in the loop to get the
right behaviour

while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {

.... where notyeteof is set in the body of the loop as follows :...

notyeteof=false;
if (sis.ttype!=sis.TT_EOF) {
sis.nextToken();
if (sis.ttype==sis.TT_EOF) {notyeteof=true;}
.... more statements .........
}

}
 
J

John B. Matthews

 (e-mail address removed) wrote:
[...]
I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
Within the loop I am using sis.nextToken() and sis.pushBack()
Now, if the eof is reached and pushedBack,

The "next call to the nextToken method" will again "return the current
value in the ttype field," TT_EOF.

will the loop continue one more time?

It shouldn't.
If pushBack() happens properly, I think the loop should continue one
more time.

It might be a useful feature, but it's not the documented behavior.

[...]

Many thanks for your feedback John ! Much appreciated.

I had to put in an additional OR condition in the loop to get the
right behaviour

while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {
[...]

Excellent! Assuming correct initialization, the predicate may be
simplified to:

while (sis.ttype != StreamTokenizer.TT_EOF || notYetEOF) {

I have to say this still seems counterintuitive. What's left to do after
seeing TT_EOF?
 
E

enokacorea

 (e-mail address removed) wrote:
[...]
I am looping while (sis.ttype != StreamTokenizer.TT_EOF)
Within the loop I am using sis.nextToken() and sis.pushBack()
Now, if the eof is reached and pushedBack,
The "next call to the nextToken method" will again "return the current
value in the ttype field," TT_EOF.
<http://java.sun.com/javase/6/docs/api/java/io/StreamTokenizer.html#pu...()>
will the loop continue one more time?
It shouldn't.
If pushBack() happens properly, I think the loop should continue one
more time.
It might be a useful feature, but it's not the documented behavior.
[...]
Many thanks for your feedback John ! Much appreciated.
I had to put in an additional OR condition in the loop to get the
right behaviour
while ((sis.ttype != StreamTokenizer.TT_EOF)||(notyeteof==true)) {

[...]

Excellent! Assuming correct initialization, the predicate may be
simplified to:

while (sis.ttype != StreamTokenizer.TT_EOF || notYetEOF) {

I have to say this still seems counterintuitive. What's left to do after
seeing TT_EOF?

--
John B. Matthews
trashgod at gmail dot com
home dot woh dot rr dot com slash jbmatthews- Hide quoted text -

- Show quoted text -

You see, if I push back something near the end of a loop I want it
processed at the start of the next loop. If TT_EOF is not set
correctly the last token will not be processed correctly.

This is what I tried to resolve with the logic I described earlier.

I am still having my original problem though !

Trying to debug this I tried a small test and found that - nextToken()
immediately followed by pushBack() does not give the same result as
not having both statements !!

Any thoughts on this ?
 
J

John B. Matthews

(e-mail address removed) wrote:
[...]
I have to say this still seems counterintuitive. What's left to do after
seeing TT_EOF?
[...]
You see, if I push back something near the end of a loop I want it
processed at the start of the next loop. If TT_EOF is not set
correctly the last token will not be processed correctly.

This is what I tried to resolve with the logic I described earlier.

I am still having my original problem though !

Trying to debug this I tried a small test and found that - nextToken()
immediately followed by pushBack() does not give the same result as
not having both statements !!

Any thoughts on this ?

Yes, the effect of pushBack() is limited to giving the same result from
nextToken(); nval and sval are unchanged. The mechanism seems intended
to provide inexpensive, one-token look-ahead, rather than unlimited undo.

<code>
import java.io.*;
public class Tokenizer {
public static void main(String[] args) throws IOException {
StreamTokenizer tokens = new
StreamTokenizer(new StringReader("Test"));
int token;
token = tokens.nextToken();
print(tokens, token);
token = tokens.nextToken();
print(tokens, token);
tokens.pushBack();
token = tokens.nextToken();
print(tokens, token);
}
private static void print(StreamTokenizer tokens, int token) {
System.out.println((token == StreamTokenizer.TT_EOF)
+ " " + tokens.sval);
}
}
</code>
<console>
false Test
true null
true null
</console>

Here's a more elaborate example:

<http://sites.google.com/site/drjohnbmatthews/enumerated-functions>

Notice how encountering TT_EOF (defined as Symbol.END) signals
successful parsing; any other result represents a (not otherwise
identified) syntax error.
 
E

enokacorea

[...]




I have to say this still seems counterintuitive. What's left to do after
seeing TT_EOF?
[...]
You see, if I push back something near the end of a loop I want it
processed at the start of the next loop. If TT_EOF is not set
correctly the last token will not be processed correctly.
This is what I tried to resolve with the logic I described earlier.
I am still having my original problem though !
Trying to debug this I tried a small test and found that - nextToken()
immediately followed by pushBack() does not give the same result as
not having both statements !!
Any thoughts on this ?

Yes, the effect of pushBack() is limited to giving the same result from
nextToken(); nval and sval are unchanged. The mechanism seems intended
to provide inexpensive, one-token look-ahead, rather than unlimited undo.

<code>
import java.io.*;
public class Tokenizer {
    public static void main(String[] args) throws IOException {
        StreamTokenizer tokens = new
            StreamTokenizer(new StringReader("Test"));
        int token;
        token = tokens.nextToken();
        print(tokens, token);
        token = tokens.nextToken();
        print(tokens, token);
        tokens.pushBack();
        token = tokens.nextToken();
        print(tokens, token);
    }
    private static void print(StreamTokenizer tokens, int token) {
        System.out.println((token == StreamTokenizer.TT_EOF)
            + " "  + tokens.sval);
    }}

</code>
<console>
false Test
true null
true null
</console>

Here's a more elaborate example:

<http://sites.google.com/site/drjohnbmatthews/enumerated-functions>

Notice how encountering TT_EOF (defined as Symbol.END) signals
successful parsing; any other result represents a (not otherwise
identified) syntax error.

--
John B. Matthews
trashgod at gmail dot com
home dot woh dot rr dot com slash jbmatthews- Hide quoted text -

- Show quoted text -

Many thanks once again for the code snippet and reference.
The snippet of course works as expected.

My issue is that I thought my code should give the identical output
either if
a) I insert a readToken() immediately followed by a pushBack() or
b) with neither of these statements
(all else being the same).

I find that this is not the case.

After further debugging I removed

while (sis.ttype != StreamTokenizer.TT_EOF) and put in a

while (!EOF)

where EOF is set if (sis.ttype == StreamTokenizer.TT_EOF)
immediately after the "chief" nextToken() - which is not subject to
pushBack().

This solved the problem !!

By the way, I visited your web site and was intrigued to find that you
are a doctor of medicine in addition to being a computer expert !

Chris
 
J

John B. Matthews

(e-mail address removed) wrote:
[...]
My issue is that I thought my code should give the identical output
either if a) I insert a readToken() immediately followed by a
pushBack() or b) with neither of these statements (all else being the
same). I find that this is not the case.

I am unable to reproduce what you describe:

<code>
import java.io.*;
public class Tokenizer {
public static void main(String[] args) throws IOException {
StreamTokenizer tokens = new
StreamTokenizer(new StringReader("Test 123"));
int token;
token = tokens.nextToken();
print(tokens, token);

token = tokens.nextToken();
print(tokens, token);

tokens.pushBack();
tokens.pushBack();
tokens.pushBack();
token = tokens.nextToken();
tokens.pushBack();
token = tokens.nextToken();
tokens.pushBack();
token = tokens.nextToken();
print(tokens, token);

token = tokens.nextToken();
print(tokens, token);
}
private static void print(StreamTokenizer tokens, int token) {
System.out.println((token == StreamTokenizer.TT_EOF)
+ " " + tokens.sval + " " + tokens.nval);
}
}
</code>
<console>
false Test 0.0
false null 123.0
false null 123.0
true null 123.0
After further debugging I removed

while (sis.ttype != StreamTokenizer.TT_EOF) and put in a

while (!EOF)

where EOF is set if (sis.ttype == StreamTokenizer.TT_EOF) immediately
after the "chief" nextToken() - which is not subject to pushBack().

This solved the problem !!

Excellent! I would also develop tests to ensure that your code rejects
invalid input.
By the way, I visited your web site and was intrigued to find that you
are a doctor of medicine in addition to being a computer expert !

I'm no expert, but sometimes I apply software engineering principles to
medical informatics. I rarely appear on TV. :)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top