StringTokenizer Lab

[Andrew Thompson]

I made a configurable 'StringTokenizer Lab'.
Comments welcome, especially on the (..Java) code.
<http://www.physci.org/token.jsp>

 

[Liz]

I made a configurable 'StringTokenizer Lab'.
Comments welcome, especially on the (..Java) code.
<http://www.physci.org/token.jsp>

--
Andrew Thompson
http://www.PhySci.org/ Open-source software suite
http://www.PhySci.org/codes/ Web & IT Help
http://www.1point1C.org/ Science & Technology

Lab is a kind of dog isn't it? Golden or Black?

 

[Andrew Thompson]

..Lab is a kind of dog isn't it? Golden or Black?

Specially bred, of course. Still took a
great deal of effort to teach it how to
tokenize strings, I can tell you.

Maybe I'll try a RegEX Alsation next.. ;-)

 

[Ryan Stewart]

Lab is a kind of dog isn't it? Golden or Black?

I don't know that I've heard it referred to as a golden lab. A golden
retriever, sure, but I'm telling you up front it's a fraud. If they really
retrieved gold, I'd sure buy one.

 

[Chris Smith]

I made a configurable 'StringTokenizer Lab'.
Comments welcome, especially on the (..Java) code.
<http://www.physci.org/token.jsp>

My one comment is this. Simple uses of StringTokenizer aren't that
difficult; the interesting bits come from use of the nextToken(String)
method. Not that I'd ever encourage anyone to use that method, but it
might be interesting to add a step-by-step version that allows changing
the delimiter in the middle of tokenizing.

If you did this and could find a way, it would be *very* interesting to
be allowed to try that last with several different API versions, so you
could observe the incompatible behavior.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[Andrew Thompson]

My one comment is this. Simple uses of StringTokenizer aren't that
difficult; the interesting bits come from use of the nextToken(String)
method. Not that I'd ever encourage anyone to use that method,

Why not, and is ther then, any more than
theoretical reasons to test/demonstrate it?*

..but it
might be interesting to add a step-by-step version that allows changing
the delimiter in the middle of tokenizing.

* ..I have used lots of StringTokenizers,
but never had the need to change one
on the fly..

If you did this and could find a way, it would be *very* interesting to
be allowed to try that last with several different API versions, so you
could observe the incompatible behavior.

...I'm not familiar with the API differences
to which you refer. Could this test be done
by simply alternating between the characters
of the supplied delimiter?

E.G. ..

<hackish vague waving of arms>

String delim = ";|,"
StringTokenizer st = new
StringTokenizer(delim.substring(0,1))
for (ii=1...) {
..
st.nextToken(delim.substring(ii,ii+1))..
}

</hackish vague waving of arms>

 

[Chris Smith]

Why not, and is ther then, any more than
theoretical reasons to test/demonstrate it?*

Guess not. Just that I am failing to see the point of code that only
demonstrates the obvious behavior, so I thought I'd throw out an idea.

..I'm not familiar with the API differences
to which you refer.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4338282

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[Andrew Thompson]

Andrew Thompson wrote: (C.S.)

Guess not. Just that I am failing to see the point of code that only
demonstrates the obvious behavior,

To be able to formulate a _link_ that
expresses (and demonstrates) that
obvious answer, that is the basic idea.

That which is obvious to you and me
and 97 others may not be obvious to
1-in-100 people. It is for them. [ ;-) ]

Also, I wanted to come up with a few
bits of easy 'this is what it works like'
type code, one for StringTokenizer..
a configurable Regex Lab is next.

...so I thought I'd throw out an idea.

Thanks! Especially since you were actually
discussing Java, and there was no mention
of dogs ... ;-)

...
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4338282

Uuuggh! That Sun log-in always gets me,
I must have three 'accounts' set-up with
them now, but cannot get _any_ of them
to work..

10 word summary for those who
(need to see a StringTokenizer
work to understand i.. ..errr,)
cannot get access to Sun?

 

[Chris Smith]

Uuuggh! That Sun log-in always gets me,
I must have three 'accounts' set-up with
them now, but cannot get _any_ of them
to work..

10 word summary for those who
(need to see a StringTokenizer
work to understand i.. ..errr,)
cannot get access to Sun?

Basically, it deals with code like:

StringTokenizer st = new StringTokenizer("test:1,2", ":");
System.out.println(st.nextToken());
System.out.println(st.nextToken(","));

IIRC, in versions prior to 1.3, the colon delimiter is consumed by the
first nextToken operation. In versions 1.3 and above, it is not, and
the second token returned is ":1".

Okay, that was more than 10 words.

--
www.designacourse.com
The Easiest Way to Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation

 

[Andrew Thompson]

So I would question whether there is any benefit to a tutorial
for newbies when we should be telling them to not use the class.

What would you recommend for pre-1.4
code where the delimiter is not changed
during tokenization?

 

[Andrew Thompson]

As I said there are plentyof third-party replacements for
StringTokenizer

There may well be.. but I cannot see
it mentioned in this thread.. Are
you thinking of some other thread perhaps?

..that are more capable and can handle the often
requested case of CSV files. A google search should turn up a
plethora.

Given both your and Chris' examples involved
changing the delimiter mid-tokenization, I fail
to se how they relate in the slightest to the
case I outline.

I think it would be more profitable
to tell the noobs.
a) Use String.split() from 1.4 if possible, otherwise
b) StringTokenize *without* stuffing with the
delimiter in the middle or the process.*

* with a link to the error at Sun
...and a Google search for
'StringTokenizer replacement'
at the end.. ;-)

 

[Andrew Thompson]

I think it would be more profitable
to tell the noobs. ..

<http://www.physci.org/token.jsp#discussion>

 

[Andrew Thompson]

...I wanted to come up with a few
bits of easy 'this is what it works like'
type code, one for StringTokenizer..
a configurable Regex Lab is next.

<http://www.physci.org/split.jsp>

 

[Dale King]

Hello, Chris Smith !

Basically, it deals with code like:

StringTokenizer st = new StringTokenizer("test:1,2", ":");
System.out.println(st.nextToken());
System.out.println(st.nextToken(","));

IIRC, in versions prior to 1.3, the colon delimiter is consumed by the
first nextToken operation. In versions 1.3 and above, it is not, and
the second token returned is ":1".

Okay, that was more than 10 words.

I got bit by that one by myself, which led me to file bug
4645089, which basically said that since they changed the
behavior they have essentially made the method worthless because
you can't know what behavior the VM will have. I said they might
as well deprecate the method.

They didn't deprecate it officially, but beginning in 1.4 the
javadocs say that StringTokenizer is only there for supporting
legacy code and is not recommended for new code.

And considering the new behavior, I don't think there is anyway
to actually use it in a beneficial way. I can't think of any real
world application where you would actually want the delimiter(s)
returned and there really isn't a clean, simple workaround to
remove the delimiters at the beginning in general.

To show what I mean about the difficulty in stripping it consider
this case:

StringTokenizer st = new StringTokenizer(s, ": ");
t1 = st.nextToken();
t2 = st.nextToken(" ,"));
t3 = st.nextToken();

I changed Chris' examle by having sace also a delimiter. If I run
this on "test : 1, :w" it works as expected before 1.3, but in
1.3 you get "the colon as the second token.

And if the string didn't have the spaces you have the additional
complication that you should only strip the colon from the first
entry in the comma separated list which complicates any loop to
read them.

While you can come up with solutions they are generally more
complex than just doing the whole scanning job yourself without
the "aid" of StringTokenizer.

All this is on top ofall of its failings with respect to handling
CSV files.

So my general advice is the same as Sun's, avoid the class
altogether . Use the String.split method which is much more
powerful or there are better replacements which work better out
there.

So I would question whether there is any benefit to a tutorial
for newbies when we should be telling them to not use the class.

 

[Dale King]

Hello, Andrew Thompson !

class.

What would you recommend for pre-1.4
code where the delimiter is not changed
during tokenization?

As I said there are plentyof third-party replacements for
StringTokenizer that are more capable and can handle the often
requested case of CSV files. A google search should turn up a
plethora.