Shall We Execute Our Automation Test case in Random Order??

0
228
UFT
UFT
Spread the love

How to create test suite for Automation?
How to create a test suite for Automation?

Automation test case design, pattern and best practices

For automation test cases, I always try to reuse what is already available in my test inventory. Always follow do not repeat yourself technique.  If at all components (functions, sub, procedure, test data, etc) are not matching with my requirement, I would create a new one.  Even there is a tendency to reuse the bunch of steps or bunch of tests even test suite itself. Like me so many like-minded persons available in the information technology industry for sure.. 😀

But as per testing best practice goes, we really need to improve our way of test development. As per that the best practice, we need to make test case independent to each other. There should not be any sharing between two test cases including test data.

Let us see with an example, what I am trying to venture through this post.

An example…
Let’s say, I have to update a previously created employee who is attached to an organization.

Automation test case

So what are the manual test case steps:

  • Create an Organization
  • Search for the same Organization and Create an Employee
  • Find the employee and find the organization and attach Employee to Organization
  • Now Update the same employee

When I will automate the same flow I will create the structure like-

Testcase1: Create Organization
Testcase2: Create Employee
Testcase3: Attach Employee to Organization
Testcase4: Update Employee

Are test cases written in Big Ball of Mud test pattern correct?

With the advancement of testing, This is one of the basic test patterns where test assets are heavily shared between the test flows.

Master driver or test suite can be also shared along with test data. There is no formal boundary or structure maintained to distinguish between components[may be test data, sub, macro, code, etc]. Assume a scenario where a component of a test asset is reused by some hundred of test flows.

Automation test case design

Now to accommodate a new test case or a set of test cases we need to change the component a little bit. From the structural point of view as master drivers or test, suites are shared, so we really do not know the impact of changing one component. With this approach, it is very tough to find out which components use all components. The change might fail some unnecessary test scripts in another area.

So If we need to randomize the test cases designed in this pattern, this might trigger failures. It will be needed for patchwork and workaround. So not so efficient way to execute them in a random manner.

Are test cases written in spaghetti test pattern correct?

We will bunch these test cases into a batch called test suite and try to execute the same.
For this case, we can simply put everything in flow. Record playback like mechanism. Or we can divide the entire flow to sub flows[test cases] and stitch them.

READ  How to Get All The Web Elements and Print Each Web Element Count In Selenium

Automation test case and spaghetti test pattern

With this approach, each flow heavily dependent on the previous flow. As the state of the application is used by the next test case, so we can not run them individually. They are tightly coupled together. In such a case, the design pattern is the spaghetti test pattern. So all test inventory which is having record playback framework and test cases are tightly coupled, we can not execute them in random order.

Are test cases are written in Chain Linked test pattern correct?

Now, where is Create Organization test case in inventory?? It is inside the Organization module. Same like Create Employee will be inside the Employee module and updating an employee may be in a transaction or some other module.

Automation test case and chain linked test pattern

These type of approach where we will have different flows coming from different modules are upgraded from the spaghetti test pattern. Each test case is considered as chain and bunch of chain hardly share the test data. But still, it depends on the execution order to get the same state of application for the test cases. From the structure, it is different from the spaghetti model. Here the test cases are shared but the master driver or the test suite or the driver is independent of each other. Maintenance of this type of automation asset and test data is a pain area for an organization.

Mostly keyword driven framework, data-driven framework, Modular decomposition approach, Function driven approach, Data oriented function-driven approach follow chain linked test pattern. So randomize the test case execution may not suite for them.

The above three approaches talk about more reusability rather than independence of test cases.

public class updateEmployee{
@Test public void Create_Organization()
{
//Creation of org code goes here
}
@Test public void Create_Employee()
{
//Creation of Employee code goes here
}
@Test public void Update_Employee()
{
//Updating of Employee code goes here
}
}

With these kinds of the approach, the Test1 becomes the precondition for Test2 and Test1 and Test2 will both become the precondition for Test3. These test cases cannot be picked individually rather they need to be run in a batch.

Automate test case

Advantages

  • No need to repeat the code.Optimum writing but maximum usage of test code
  • Structured or semi-structured so finding out the batch and execute them is easy.
  • Mostly follows “do not repeat yourself” logic hence become modular with great ROI of automation.
  • Less execution time.
Automation test case

DisAdvantages

  • These type of test pattern mostly fail to give correct failure count. Or it is a constant pain for the test engineer to keep track. How? Let me explain… In the above test, flow says Testcase-1[Create Organization] passed but Test Flow-2 [Employee Creation] failed. As Test case-1 and Test case-2 are preconditions for Testcase-3, Hence Testcase-3 also will fail. or may not get executed depending upon your test configuration settings. So Report may look like Test-3 Passed-1 Failed-2. But no way, we can run these 2 failures. We need to execute 3 test cases once again. In an organization, in test inventory, thousands of scripts run. And if we consider ten parentage failures. We will have issues in reporting how much did we run really to get all test cases passed. Engineers need to be extremely careful to report
  • There is no way you can pick the failed test cases and execute them. As a result execution time becomes high. It becomes unreliable too
  • There is a huge chance that the polluted or corrupted test data may enter into the next flow and may fail the flow.
READ  How To Resolve OSError : WinError 6 The Handle is Invalid

Are test cases written in Hermetic test pattern correct?
Hermetic test pattern put more focus on test independence rather reusability. As per this test pattern each test case should be independent of each other. No sharing of test assets between test cases batched even in the single test suite. Let us see how it will look like when we design our test cases as per the Hermetic test pattern.
Test Name-CreateOrganization

public class CreateOrganization{
@precondition
//precondition steps go here
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}

Another Test Name-CreateEmployee

public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link in between
}
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
public void Delete_Organization()
{
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}
}

Test Name-UpdateEmployee

public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link/relation in between
}
@precondition
public void Create_Employee()
{
//Employee Creation code goes here
}
@Test
public void Update_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test.
//Application goes back to original condition.
public void Delete_Organization()
{
}
public void Delete_Employee()
{
}
}
Automation test case

Advantages

  • Fully Structured and independent to each other so no problem to pick up any test cases and execute.
  • No polluted or corrupted test data propagation between tests
  • Uniquely identify the issues with test data
  • Easy reporting and rerun report generation.
Disadvantages
  • A lot of refactoring required if we have a large test inventory prepared in another model
  • Need to prepare test data for atomic test cases which is a hectic job.
  • Execution time increases by multiple times.

So we need to take a conscious call if we will follow the hermetic design or batch mode test design. Mainly if we test a flow based scenario it is always better to go for batch mode test design. Otherwise, we must follow an atomic test pattern or hermetic test pattern.

With hermetic test pattern, we will be able to randomize our test execution.
I always try to make test cases independent. If not possible or if that does not make sense, I go for batch mode or bunch mode and tag them differently.
Mostly Smoke testcases[a set of test cases running day and night to make sure build is stable and correct], I prefer them independent and execute them in random order.

Choosing other factors for test cases suite generations

Below are the factors that need to be checked when creating an effective test suite…

  1. On which platform we are getting more bugs?
  2. Will we miss bugs if we cut the automation?
  3. Which test discovers most bugs?
  4. Test that uncovers the related bugs but not part of the nightly run
  5. Which hardware to use?
  6. Shall we use hardware 24*7?
  7. shorter time scale from where the bug got introduced and caught?
  8. How much to reuse?
  9. How many tests cases leverage is best?
  10. Is there any report analysis?
READ  What Are the Different Types of Test Management Tool

How to design your automation test regression suite?

Apart from the above way of designing the test cases, the regression suite should also have an easy interface to select the test cases in groups, individually or any sequence of choice. We need to make sure no test can influence another test (just now, we have discussed). The simplest way to achieve this is to start test cases from the beginning and alternatively start from the end.

Implementation of precondition and postcondition should be part of the test case itself.

Life Cycle of automation test cases

automation Testcase life cycle
automation Testcase life cycle

All active development must go through a test case review mechanism for efficiency and coverage.

all reviewed automation test cases must be sent for execution as a candidate for automation.

All candidates who run successfully becomes the active set of the regression suite. and those cases which fail for scripting reason go to development stage to revalidate.

If the product has changed and the test is no longer valid, it goes to the suspended stage.

With all periodic maintenance schedule, these test cases must be retired and removed.

One suite for all or multiple tests for different testing?

I do not personally like the idea of creating one suite for all kind of testing. In my view, we need to prepare different test suits for different type of testing.

Nightly or daily regression suits are having the most shorter execution time. They run for a shorter or limited period of time and provide quick feedback.

Different regression suits
Different regression suits

A weekly regression suite is the most important flows that need to cover or a must to include flows. Apart from these, we can have functionality wise regression suite.

We can include a meta tag or meta name for tagging purpose. The project team can schedule runs as per their need.

Tagging the test cases for a better track

Tagging a function/component or giving a clear description can stop code duplication. Automation test engineers should look for the inventory before starting a code/function.

Similarly, naming convention plays a very important role. More the clear the naming it is more easy to track the test cases.

Test cases around Hotfix

A hotfix is a piece of code that is put into the production system very quickly to stop further issues. It is always advisable to create and execute test cases related areas before we put the code to production. These test cases also should be part of your Automation suite.

Conclusion:

By adding multiple machines (physical or virtual) we can speed up automation by running independent test cases parallel.so our test cases need to be created and structured in such a way that they can be used in any conditions

  • On the fly selection
  • sequentially
  • Parallel

Let me which technique you used for your automation test suite preparation by commenting below.

image Credit:pro-it-service.com


Spread the love
I am the founder and owner of the blog - TechTravelHub.com, always love to share knowledge on test automation,tools,techniques and tips.I am a passionate coder of Java and VBScript.I also publish articles on Travel ideas and great honeymoon destinations.Apart from these, I am a gear-head,love to drive across India. I have shared lots of articles here on How to travel several parts of India.Customization of cars aka car modification is my another hobby.Get in touch with me on ani01104@gamil.com

LEAVE A REPLY

Please enter your comment!
Please enter your name here