To perform tokenization in java is pretty easy.
Suppose we want to tokenize this sentence”I am going to delhi”. And we want output like as follow:
I
am
going
to
delhi
The code is;
String s=”I am going to delhi”;
StringTokenizer st = new StringTokenizer(s);
while (st.hasMoreTokens())
{
System.out.println(st.nextToken())
}
See also: