Class StringTokenizer in Java

digitization, transformation, binary

StringTokenizer in Java:

The StringTokenizer class implements a simple, delimiter-based string tokenizer. In other words, a StringTokenizer is used to break a String into tokens separated by delimiter characters. By default, the class uses the standard whitespace characters as delimiters, but we can specify any delimiter characters we want, either when the StringTokenizer is created or on a token-by-token basis. We can also specify whether the delimiters are returned as tokens or not.

A StringTokenizer returns the tokens in its String in order, either as String objects or as Objects in an Enumeration.StringTokenizer shouldn’t be confused with the more complex java.io.StreamTokenizer,which parses a stream into tokens that are similar to those used in the Java language.

Ads code goes here

The StringTokenizer class allows us to break a string into tokens with the specified characters as a delimiter. The tokenization method is much simpler than the one used by the StringTokenizer class.

The StringTokenizer class is also used to recover the words in multiword String. It is used for reading the input. To use StringTokenizer we need to import java.util.StringTokenizer class to our code.

The StringTokenizer does not distinguish among identifiers, numbers and quoted Strings. It also does not recognize and skip comments. The set of delimiters (the characters that separate tokens) may be specified either at creation time or on a per-token basis.

For an example of StringTokenizer behaves in one of the two ways, depending on whether it was created with the return tokens flag. This flag can have two values true or false.
StringTokenizer class implements the Enumeration interface hence we can access elements with hasMoreElements() and nextElement() methods.

READ  What is a Compilation Unit in Java?

In addition, It is often useful to break a set of text apart into tokens, where the characters separating the tokens can be specified.

The StringTokenizer class provides that. We can create an instance of the StringTokenizer class using a String and an optional second String as a separator.

Once it is done, we can use hasMoreTokens() method and nextToken() methods to fetch the tokens from the String.

When we create a StringTokenizer, we can specify a string of delimiter characters to use for the entire string or we can use the default delimiter as whitespace.

We can also specify if the delimiters themselves should be returned as tokens or not.

Token

The individual words after StringTokenizer generates are called Tokens.

Case-1:: If the flag is false:

If the flag is false, delimiters’ characters merely serve to separate tokens.

A token is a maximal sequence of consecutive characters that are delimiters.

Case-2:: If the flag is true:

If the flag is true, delimiters’ characters are considered to be tokens.

A token is either one delimiter character or a maximal sequence of consecutive characters that are not delimiters.

How to work:

StringTokenizer  st=new StringTokenizer("I love Java");
while(st.hasMoreToken())
{
System.out.println(st.nextToken());
} 

The output of the code is:
I
love
Java

When the default separator is blank or white space StringTokenizer class allows us to provide our own separator. For this, we need to call another version of the StringTokenizer constructor with the second argument as a separator.

StringTokenizer  st=new StringTokenizer("I(tab)love(tab)Java","\t");
while(st.hasMoreToken())
{
System.out.println(st.nextToken());
} 

The output of the code is:
I
love
Java

The structure of class StringTokenizer is given as:

public class java.util.StringTokenizer extends java.lang.Object implements java.util.Enumeration{
//constructors:
public StringTokenizer(String str);//constructs a StringTokenizer for the specified String str.The tokenizer uses the default delimiter set,which is "tnr",the space character,the tab character,the newline character and the carriage return character.
public StringTokenizer(String str,String delimiter);//constructs a StringTokenizer for the specified String str.The character specified in the argument as String delimiter is the delimiter for separating tokens.
public StringTokenizer(String str,String delimiter,boolean returnTokens);//constructs A StringTokenizer for the specified String str.The character specified in the argument as String delimiter is the delimiter for separating tokens.If the return Tokens flag is true,then the delimiter characters are also returned as Tokens.Each delimiter characters are also returned as a String of length one.If the flag is false,the delimiter characters are skipped and only serve as separators between tokens.
//Methods:
public int countTokens();
public boolean hasMoreElements();
public boolean hasMoreTokens();
public Object nextElement();
public String nextToken();
public String nextToken(String delimit)
} 

The details of the class structure are given as follows:

public StringTokenizer(String str);

public StringTokenizer(String str) method constructor creates a StringTokenizer that breaks apart the given string using the default delimiter characters. The default delimiters are the standard whitespace characters: space, tab (\t), carriage return (\r), and newline (\n).

READ  Abstract Class SocketImpl in Java

Parameter
str – The String to be tokenized.

public StringTokenizer(String str, String delim);

public StringTokenizer(String str, String delim) constructor creates a StringTokenizer that breaks apart the given string using the characters in delim as delimiter characters.

Parameter
str – The String to be tokenized.
delim – The delimiter characters.

public StringTokenizer(String str, String delim, boolean returnTokens);

public StringTokenizer(String str, String delim, boolean returnTokens) constructor creates a StringTokenizer that breaks apart the given string using the characters in delim as delimiter characters. If returnTokens is true, the delimiters are returned as tokens; otherwise they are skipped.

Parameter
str – The String to be tokenized.
delim – The delimiter characters.
returnTokens – A boolean value that indicates whether or not delimiters should be returned as tokens.

public int countTokens();

public int countTokens() method returns the number of tokens that remain in the string, which is the same as the number of times nextToken() can be called before an exception is thrown.

This method returns the number of tokens remaining in the string.

public boolean hasMoreElements();

public boolean hasMoreElements() method returns the result of calling hasMoreTokens().

This method returns true if there are more tokens to be returned; false otherwise.

public boolean hasMoreTokens();

public boolean hasMoreTokens() method returns true if this object has more tokens to return.

This method returns true if there are more tokens to be returned; false otherwise.

public Object nextElement();

public Object nextElement() method returns result of calling nextToken().

This method returns the next token as an Object.

public String nextToken();

public String nextToken() method returns the next token.

READ  Hire Java Builders With Working experience in Top 10 Java Frameworks

This method returns the next token as a String.

public String nextToken(String delim);

public String nextToken(String delim) method sets the delimiter characters to the characters in the given string and then returns the next token. The change in delimiters persists until another call to this method changes them again.

This method returns the next token as a String.

Parameter
delim – The delimiter characters.

Apart from these StringTokenizer class also has inherited methods from class- Object. They are as follows:

  • clone()
  • finalize()
  • hashCode()
  • notifyAll()
  • wait()
  • wait(long, int)
  • equals(Object)
  • getClass()
  • notify()
  • toString()
  • wait(long)

Share and Enjoy !

Leave a Comment

Your email address will not be published. Required fields are marked *