Shall We Execute Our Automation Test case in Random Order??

0
33
Spread the love
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

How to create test suite for Automation?
How to create a test suite for Automation?

Automation test case design, pattern and best practices

For automation test cases, I always try to reuse what is already available in my test inventory. Always follow do not repeat yourself technique.  If at all components (functions, sub, procedure, test data, etc) are not matching with my requirement, I would create a new one.  Even there is a tendency to reuse the bunch of steps or bunch of tests even test suite itself. Like me so many like-minded persons available in the information technology industry for sure.. 😀

But as per testing best practice goes, we really need to improve our way of test development. As per that the best practice, we need to make test case independent to each other. There should not be any sharing between two test cases including test data.

Let us see with an example, what I am trying to venture through this post.

An example…
Let’s say, I have to update a previously created employee who is attached to an organization.

Automation test case

So what are the manual test case steps:

  • Create an Organization
  • Search for the same Organization and Create an Employee
  • Find the employee and find the organization and attach Employee to Organization
  • Now Update the same employee

When I will automate the same flow I will create the structure like-

Testcase1: Create Organization
Testcase2: Create Employee
Testcase3: Attach Employee to Organization
Testcase4: Update Employee

Are test cases written in Big Ball of Mud test pattern correct?

With the advancement of testing, This is one of the basic test patterns where test assets are heavily shared between the test flows.

Master driver or test suite can be also shared along with test data. There is no formal boundary or structure maintained to distinguish between components[may be test data, sub, macro, code, etc]. Assume a scenario where a component of a test asset is reused by some hundred of test flows.

Automation test case design

Now to accommodate a new test case or a set of test cases we need to change the component a little bit. From the structural point of view as master drivers or test, suites are shared, so we really do not know the impact of changing one component. With this approach, it is very tough to find out which components use all components. The change might fail some unnecessary test scripts in another area.

So If we need to randomize the test cases designed in this pattern, this might trigger failures. It will be needed for patchwork and workaround. So not so efficient way to execute them in a random manner.

Are test cases written in spaghetti test pattern correct?

We will bunch these test cases into a batch called test suite and try to execute the same.
For this case, we can simply put everything in flow. Record playback like mechanism. Or we can divide the entire flow to sub flows[test cases] and stitch them.

Automation test case and spaghetti test pattern

With this approach, each flow heavily dependent on the previous flow. As the state of the application is used by the next test case, so we can not run them individually. They are tightly coupled together. In such a case, the design pattern is the spaghetti test pattern. So all test inventory which is having record playback framework and test cases are tightly coupled, we can not execute them in random order.

Are test cases are written in Chain Linked test pattern correct?

Now, where is Create Organization test case in inventory?? It is inside the Organization module. Same like Create Employee will be inside the Employee module and updating an employee may be in a transaction or some other module.

Automation test case and chain linked test pattern

These type of approach where we will have different flows coming from different modules are upgraded from the spaghetti test pattern. Each test case is considered as chain and bunch of chain hardly share the test data. But still, it depends on the execution order to get the same state of application for the test cases. From the structure, it is different from the spaghetti model. Here the test cases are shared but the master driver or the test suite or the driver is independent of each other. Maintenance of this type of automation asset and test data is a pain area for an organization.

Mostly keyword driven framework, data-driven framework, Modular decomposition approach, Function driven approach, Data oriented function-driven approach follow chain linked test pattern. So randomize the test case execution may not suite for them.

The above three approaches talk about more reusability rather than independence of test cases.

public class updateEmployee{
@Test public void Create_Organization()
{
//Creation of org code goes here
}
@Test public void Create_Employee()
{
//Creation of Employee code goes here
}
@Test public void Update_Employee()
{
//Updating of Employee code goes here
}
}

With these kinds of the approach, the Test1 becomes the precondition for Test2 and Test1 and Test2 will both become the precondition for Test3. These test cases cannot be picked individually rather they need to be run in a batch.

Automate test case

Advantages

  • No need to repeat the code.Optimum writing but maximum usage of test code
  • Structured or semi-structured so finding out the batch and execute them is easy.
  • Mostly follows “do not repeat yourself” logic hence become modular with great ROI of automation.
  • Less execution time.
Automation test case

DisAdvantages

  • These type of test pattern mostly fail to give correct failure count. Or it is a constant pain for the test engineer to keep track. How? Let me explain… In the above test, flow says Testcase-1[Create Organization] passed but Test Flow-2 [Employee Creation] failed. As Test case-1 and Test case-2 are preconditions for Testcase-3, Hence Testcase-3 also will fail. or may not get executed depending upon your test configuration settings. So Report may look like Test-3 Passed-1 Failed-2. But no way, we can run these 2 failures. We need to execute 3 test cases once again. In an organization, in test inventory, thousands of scripts run. And if we consider ten parentage failures. We will have issues in reporting how much did we run really to get all test cases passed. Engineers need to be extremely careful to report
  • There is no way you can pick the failed test cases and execute them. As a result execution time becomes high. It becomes unreliable too
  • There is a huge chance that the polluted or corrupted test data may enter into the next flow and may fail the flow.

Are test cases written in Hermetic test pattern correct?
Hermetic test pattern put more focus on test independence rather reusability. As per this test pattern each test case should be independent of each other. No sharing of test assets between test cases batched even in the single test suite. Let us see how it will look like when we design our test cases as per the Hermetic test pattern.
Test Name-CreateOrganization

public class CreateOrganization{
@precondition
//precondition steps go here
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}

Another Test Name-CreateEmployee

public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link in between
}
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
public void Delete_Organization()
{
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}
}

Test Name-UpdateEmployee

public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link/relation in between
}
@precondition
public void Create_Employee()
{
//Employee Creation code goes here
}
@Test
public void Update_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test.
//Application goes back to original condition.
public void Delete_Organization()
{
}
public void Delete_Employee()
{
}
}
Automation test case

Advantages

  • Fully Structured and independent to each other so no problem to pick up any test cases and execute.
  • No polluted or corrupted test data propagation between tests
  • Uniquely identify the issues with test data
  • Easy reporting and rerun report generation.
Disadvantages
  • A lot of refactoring required if we have a large test inventory prepared in another model
  • Need to prepare test data for atomic test cases which is a hectic job.
  • Execution time increases by multiple times.

So we need to take a conscious call if we will follow the hermetic design or batch mode test design. Mainly if we test a flow based scenario it is always better to go for batch mode test design. Otherwise, we must follow an atomic test pattern or hermetic test pattern.

Don't miss out!
Subscribe To Newsletter

Receive top technical news, lesson ideas, travel tips and more!

Invalid email address
Give it a try. You can unsubscribe at any time.

Spread the love
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

LEAVE A REPLY

Please enter your comment!
Please enter your name here