What is TestNG?
What is TestNG?

What is TestNG?

Introduction to TestNG

TestNG (Test Next Generation) is a testing framework developed by Cedric Beust that is inspired from JUnit and NUnit. TestNG offers more functionalities and features like-

  • It provides an upgraded execution model with no complex test suite concept. Testcases can be easily grouped, prioritized and executed.
  • It comes with data-driven test support with @DataProvider annotation and XML configurations along with its support for parameterization.
  • It provides robust and flexible test configurations with multiple instances of the same test with @Factory annotation.
  • It supports a way to test if our code is multi thread-safe.
  • It provides powerful and variety of annotations that make test case creation very easy.
  • It provides support for various parameters.
  • It provides BeanShell for further flexibility.
  • It provides a keyword named invocationcount that can be used to execute the same test cases multiple passes.
  • It allows us to run our test code in a thread pool with different policies like one thread per test or all methods in one owned thread etc. So it supports multi-threading.
  • It is available in various tools as plugins(like an eclipse, Maven, etc).
  • It uses default JDK functions for running tests and logging.
  • It supports dependent methods for application server testing.
  • It supports multiple ways to execute test cases. So users can define the dependent test cases.
  • TestNg provides a set of open APIs that helps in adding custom plugins or extensions.
  • It supports testing integrated class.
  • It separates compile time test code from runtime data and configuration.
  • It supports configuration change on demand or during runtime.
  • The annotations are clear and easy to understand.
  • It supports log generation.
  • It supports the distribution of test cases on different machines.
  • It supports cross-browser testing when attached with the browser automation tools like Selenium.

TestNG comes by default as a plugin with Eclipse, IntelliDB, and NetBeans. It also comes with Apache ANT. If we are using the Maven build system, TestNG has support for that too. The Jenkins(previously Hudson) server also has built-in support for TestNG.

Test code coverage tools like Cobertura works beautifully with TestNG. TestNG is distributed under Apache Software license and it is open-source software.

What type of testing can be done in TestNG?

We can perform various kinds of testing except performance and User acceptance testing. Also, TestNG being an automation framework, obeys the boundaries of automation testing like manual intervention or human logic or captcha code, etc. TestNG can successfully perform the following kind of testing:

  • Isolation code test
  • Unit testing
  • Smoke testing
  • Integration testing
  • Regression testing
  • End to End testing
  • Exploratory testing

A comparison with JUnit and TestNG

JUnit is often found with the popular IDEs as a default Unit test tool. Maybe this is one of the prime reasons why we get to see and hear about Junit more than TestNG. However, TestNG is not only used for Unit tests but we can use it for the other type of testing as I stated above.

Here are the prime differences between TestNG and Junit.

 

 

 

Differentiator JUnit TestNG
Annotations @BeforeClass and @AfterClass methods need to be declared as static. It is also a static programming model that lets us recompile every time. No such declaration needed. But uses java object-oriented features.
TestNG provides additional set and teardown [email protected],
@AfterSuite, @BeforeTest, @AfterTest, @BeforeGroup, and @AfterGroup,
@BeforeMethod and @AfterMethod.
Parameterization in JUnit we need to use @RunWith and @Parameters to parameterize a test.
@Parameters method should return a List[] with actual values that need to be feed into the actual test.
TestNG can feed data into the test method into two different ways
1. using testng.xml
2. using @DataProvider method
Usage Mainly used in Unit testing, but now can be used to test all kinds of testing except performance testing. It can be used various automation testing except performance testing.
Dependency Control Dependency testing is not permitted. It can be used for dependency testing.
Configuration control Configuration control is not that great. Configuration control is great.
Method naming convention We have to name the method in a particular way. There is no restriction.
Grouping No grouping concept. TestNG supports the grouping concept.
READ  Learn Quickly on How to Work With TestNG XML

 

Now our objective is through. This is an example of how to handle exceptions in TestNG.

How To Handle Sequence Error with @ Test Annotation in TestNG?

While using the TestNG framework, we provide @Test annotation for the original test method. By default, TestNG follows the order, as we mentioned in the testng.xml
Example:

public class Test1 {
  @Test
  public void testMethod1() {
  }
}

 

So the code says that testMethod1() is a function and since it is annotated with @Test so it will execute.

The problem starts when we have multiple functions to be covered as a part of the test.

@Test
  public void testMethodForRefresh()
  {
         System.out.println("Refresh the page");
  }
 @Test
  public void testMethofForClick()
  {
         System.out.println("Click on a link");
  }

our objective is to Refresh a page and click on a link. Accordingly, we have coded. But when you will execute the test cases, in the output, it is seen instead of the Refresh method, Click method got executed first then the Refresh method got executed.

Below are the ways how we can resolve this issue:

Process One:
TestNG follows the alphabetic order to pick up the methods when all the methods have the same annotation as @Test. So name all your methods accordingly.

Like – append Test00X in front of the methods. The first method what we need to call and execute will have name Test001_MethodForRefresh and next method will be Test002_MethofForClick

@Test
  public void Test001_MethodForRefresh()
  {
         System.out.println("Refresh the page");
  }
 @Test
  public void Test002_MethofForClick()
  {
         System.out.println("Click on a link");
  }

 

Process two:
A little more advanced upgrade is to set the priority of the method. like: @Test(Priority=1). In this way, we don’t have to change the method name.

@Test (Priority=1)
  public void testMethodForRefresh()
  {
         System.out.println("Refresh the page");
  }
 @Test(Priority=2)
  public void testMethofForClick()
  {
         System.out.println("Click on a link");
  }

 

Process three:
This is another approach where we need to mention the sequence in testng.xml or pom.xml. In case we want to run tests in random order we need to set the preserve-order to false.



documentation for @Test is available here: http://testng.org/doc/documentation-main.html

How to Resolve Dependency in TestNG?

Dependency in TestNG is an important concept. Dependency occurs when some set of methods needs the same set of data that needs to be shared among them. This is called dependency.

Let us see some practical examples, lets us assume that an employee joins an organization the hiring team creates the employee and the system will generate an employee id for that employee.

Now the HR team assigns some designation and calculates salary for that employee. So for them, employee id is the mandatory field to get.

Lastly, when the employee leaves the organization, the HR team again needs the employee’s id to delete the data from the database.

READ  Learn How To Install TestNG for IDEs Quickly

So for all cases, the second team and third team depend on the data generated by the first – hiring team. So in this way, they are dependant on the first team.

When we create test cases for them in order to check the second and third functionalities, we need data from the first test case.

dependency test in TestNG
dependency in TestNG

Dependency Types

There are two types of dependencies. They are as follows:

  1. Hard Dependency
  2. Soft Dependency

Dependency can be on methods or groups

  • dependsOnMethod in @Test annotation
  • dependsOnGroups in @Test annotation

Hard Dependency with dependsOnMethods

Incase if a method depended upon fails and we have a hard dependency on it by the attribute (alwaysRun= false) the method that depends on it is not called and marked as fail instead it will be marked as skipped. This skipped method will be reported in the final report. (color coding is neither red nor green)

package com.test;

import org.testng.annotations.Test;

public class DependencyTest {
	
	@Test
	public void test1()
	{
		System.out.println("This is Test1");
	}
	
	@Test(dependsOnMethods="test1",alwaysRun=false)
	public void mainTest()
	{
		System.out.println("This is Main Test");
	}
}

Here the mainTest() method is dependent on test1() method.

The testng.xml looks like-

<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd"> 
 <suite name = "MyDependencyCheck" verbose="1"> 
    <test name = "MyDependencyCheck"> 
       <classes>
            <class name="com.test.DependencyTest"/>
    </classes>
	</test>
 </suit

The execution report looks like:

Hard Dependency in TestNg
Hard Dependency in TestNg

The testNG execution report is as follows:

The TestNG execution report
The TestNG execution report

This is good as long as the base method on which our actual method is dependant is passing. Let’s test the negative scenario to check the failed cases.

Let us introduce the failure in the base method i.e in test1()

package com.test;

import org.testng.annotations.Test;

public class DependencyTest {
	
	@Test
	public void test1()
	{
		System.out.println("This is Test1");
		int h=1/0;
	}
	
	@Test(dependsOnMethods="test1",alwaysRun=false)
	public void mainTest()
	{
		System.out.println("This is Main Test");
	}
}

The execution report will look like:

Hard Dependency in TestNG for Failed case
Hard Dependency in TestNG for Failed case

The console log looks like:
[RemoteTestNG] detected TestNG version 7.0.0
This is Test1
FAILED: test1
java.lang.ArithmeticException: / by zero
at com.test.DependencyTest.test1(DependencyTest.java:11)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:584)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:172)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:804)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:145)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.testng.TestRunner.privateRun(TestRunner.java:770)
at org.testng.TestRunner.run(TestRunner.java:591)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
at org.testng.SuiteRunner.run(SuiteRunner.java:304)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
at org.testng.TestNG.runSuites(TestNG.java:1032)
at org.testng.TestNG.run(TestNG.java:1000)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:115)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

SKIPPED: mainTest
java.lang.Throwable: Method DependencyTest.mainTest()[pri:0, instance:[email protected]] depends on not successfully finished methods
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:99)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.testng.TestRunner.privateRun(TestRunner.java:770)
at org.testng.TestRunner.run(TestRunner.java:591)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
at org.testng.SuiteRunner.run(SuiteRunner.java:304)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
at org.testng.TestNG.runSuites(TestNG.java:1032)
at org.testng.TestNG.run(TestNG.java:1000)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:115)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

===============================================
Default test
Tests run: 2, Failures: 1, Skips: 1
===============================================

===============================================
Default suite
Total tests run: 2, Passes: 0, Failures: 1, Skips: 1
===============================================

As it is a hard dependency the mainTest() method depends on test1() method which is failing. Hence failure count is 1. As per the logic once the base method fails, TestNG will not execute the test1() method, this is the reason it gets skipped and showing skips count-1.

Soft Dependency with dependsOnMethods

Incase if a method depended upon fails and we have a soft dependency on it by the attribute (alwaysRun= true) the method that depends on it is called and executed marked as pass instead of skip.

Let’s change in the code to make it (alwaysRun= false).

package com.test;

import org.testng.annotations.Test;

public class DependencyTest {
	
	@Test
	public void test1()
	{
		System.out.println("This is Test1");
		int h=1/0;
	}
	
	@Test(dependsOnMethods="test1",alwaysRun=true)
	public void mainTest()
	{
		System.out.println("This is Main Test");
	}
}

Now the execution report looks like:

Soft Depenedency in TestNG for Failed case
Soft Dependency in TestNG for Failed case

The TestNG execution report looks like:

soft Dependency in TestNG
soft Dependency in TestNG

The console log

[RemoteTestNG] detected TestNG version 7.0.0
This is Test1
This is Main Test
PASSED: mainTest
FAILED: test1
java.lang.ArithmeticException: / by zero
at com.test.DependencyTest.test1(DependencyTest.java:11)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:584)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:172)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:804)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:145)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1507)
at org.testng.TestRunner.privateRun(TestRunner.java:770)
at org.testng.TestRunner.run(TestRunner.java:591)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
at org.testng.SuiteRunner.run(SuiteRunner.java:304)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
at org.testng.TestNG.runSuites(TestNG.java:1032)
at org.testng.TestNG.run(TestNG.java:1000)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:115)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

READ  What is Unit Testing?

===============================================
Default test
Tests run: 2, Failures: 1, Skips: 0
===============================================

===============================================
Default suite
Total tests run: 2, Passes: 1, Failures: 1, Skips: 0
===============================================

Here if you observe total testcases=2 passes=1 and Failure=1.

Difference between dependsOnGroups with dependsOnMethods in TestNG?

If we are using groups, there is no refactoring issue. Till the point we do not modify the dependsOnGroups or group attributes, our test can execute freely with proper dependency set up.

In case, If we need to add a new method in the dependency chain, we need to put it in the right group. We do not need to modify any other method to make sure it depends on the correct group.

How to use regular expression in TestNG?

We can use the following way to use regular expression in an expression:

dependsPnMethods={"xxx.*"}

let’s check an example:

@Test(groups={"smokeTestSlow"})
public void myInitializeMethod(){
System.out.println("My Initialize Methods");
}
@Test(dependsOnGroups="smoke.*"})
public void myTestMethod(){
System.out.println("Testing dependency");
}

Here we are declaring as depending on any group, matching with smoke.* that actually guarantees that myInitializeMethod() is always invoked before the myTestMethod().

How to Use Parameterization in TestNG?

Parameterization allows us to run a set of tests over and over using different values. There are two ways we can utilize the parameterization feature. They are as follows:

  • Using testng.xml
  • Using DataProvider

How to pass Parameters with testng.xml?

We can define parameters in the testng.xml and access them in the source code. This is how we aaccomplish Parameterization in TestNG. The test code will look like:

package com.test;

import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class ParameterTest {
	@Test
	@Parameters({"MyNumber"})
	public void ParameterTestOne(String MyNumber)
		{	
		System.out.println("the value " + MyNumber);
		}
}

The testng.xml will look like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" verbose="1">
  <test name="Test">
  <parameter name="MyNumber" value="10"/>
    <classes>
      <class name="com.test.ParameterTest"/>
      
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

The output will look like:

Parameterization in TestNG
Parameterization in TestNG

The console log is:

[RemoteTestNG] detected TestNG version 7.0.0
the value 10

===============================================
Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

TestNG report will look like

Parameterization in TestNG report
Parameterization in TestNG report

The parameter can be defined in the <suite> level as well. As <Suite> is the root tog of the testng.xml, the parameter is accessible throughout the test.

In case, we have the same parameter defined in <Suite> and <test> then the <test> tag can see the value associated with the parameter. Outside <test>  </test> tag, rest of the code will consume the   <Suite>’s  parameter.
let’s create a class with two methods. These two methods will have the same parameter name. One will consume from the suite level and the other one will consume the test level’s Parameter. This is an important concept to remember in Parameterization in TestNG.

package com.test;

import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class ParameterTest {
	@Test
	@Parameters({"MyNumber"})
	public void ParameterTestOne(String MyNumber)
		{	
		System.out.println("the value from metod one " + MyNumber);
		}
	@Test
	@Parameters({"MyNumber"})
	public void ParameterTestTwo(String MyNumber)
	{	
	System.out.println("the value from metod two " + MyNumber);
	}
}

The testng.xml will look like-

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" verbose="1">
<parameter name="MyNumber" value="20"/>
  <test name="Test">
  <parameter name="MyNumber" value="10"/>
    <classes>
      <class name="com.test.ParameterTest">
      <methods>
                    <include name="ParameterTestOne" />
       </methods>
      </class>
    </classes>
    </test>
    <test name="test2">
    <classes>
      <class name="com.test.ParameterTest">
      <methods>
                    <include name="ParameterTestTwo" />
       </methods>
      </class>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

Notice – We have not given any parameter for ParameteTestTwo method. The report will look like:

Parameterization in TestNG with two parameters
Parameterization in TestNG with two parameters

The console output will be as follows:

[RemoteTestNG] detected TestNG version 7.0.0
the value from method one 10
the value from method two 20

===============================================
Suite
Total tests run: 2, Passes: 2, Failures: 0, Skips: 0
===============================================

Testng.xml supports the following type of values- This is an important concept to remember in Parameterization in TestNG.

  • String
  • int/ Integer
  • boolean/Boolean
  • byte/ Byte
  • char/ Character
  • double/ Double
  • float/Float
  • long/Long
  • short/Short

=======================================

LEAVE A REPLY

Please enter your comment!
Please enter your name here