- 1 Introduction to What is Automation Testing
- 1.1 The objective of the automation are:
- 1.2 Few basic criteria for Automation:
- 1.3 Importance of framework –
- 1.4 Examples of automation:
- 1.5 Steps for Automation:
- 1.6 So creating a basic framework needs the following things covered:
- 1.7 Phases of Automation Testing:
- 1.8 Why does Automation fail?
- 1.9 Here are the top ten features of today’s automation.
- 1.9.1 Coverage:
- 1.9.2 Extended environment and multitool Support:
- 1.9.3 Business Process:
- 1.9.4 Easy Reporting and Self-documentation:
- 1.9.5 Reusability:
- 1.9.6 Maintain Standards:
- 1.9.7 How to put automation Standards?
- 1.9.8 Advanced features/Recovery Scenario:
- 1.9.9 Ease of debugging, logging and feedback system:
- 1.9.10 Look for better ROI:
- 1.9.11 Easy Refactoring and update:
- 1.10 Share and Enjoy !
Introduction to What is Automation Testing
Most of the time the test automation engineers build a framework or rectify the current one or upgrade the current framework.
The objective of the automation are:
- Test execution can be completed faster
- consistent and effectiveness
- Reusability makes better ROI.
- Reduce the cost of testing
Few basic criteria for Automation:
- The tool must support UAT
- Multiple releases should be carried out
- Less manual intervention
- The project should be in the spiral model.
- UAT should be stable
- the budget should be there to carry out expensive automation
Importance of framework –
- Easy Maintenance
- Better ROI
Examples of automation:
- sending bank statement
- credit salary information
- Any manual process which is repetitive and tedious.
For automation, we need to pick up a tool and script using an underlying programming language. Examples of the tools-UFT/QTP, Selenium, Silktest, RFT, Test partner.
Automation Testing is activated if there are two conditions
1. if you are doing regression testing for single or multiple datasets
2. if your application is stable enough to perform the test repeatedly
Steps for Automation:
- Collect the acceptance test cases/ regression test cases
- Record and code the script
- Improve the test script in order to reliability.
- Parameterize the hardcoded values
- Separate the test data from code by moving variables to separate files
- Create utility functions
So creating a basic framework needs the following things covered:
- Prototype components
- Test designing
- Create test interpreter
- Create AUT specific test drivers
- Add repository
- Add logger, reports capabilities
- Create test documentation.
Phases of Automation Testing:
- Development phase
- Execution phase
- Maintenance phase
Why does Automation fail?
These are the main reasons why automation fails.
- Iterative Development
- Different failure
- Technology Change
Here are the top ten features of today’s automation.
Coverage is the most important feature of any Automation success. In my view, a simple framework creation that can cover more is much more acceptable than a complex framework that covers less. So decide correctly by which approach you can get more coverage.
When we get a big flow to automate, generally seen very less time is given to make it done. Management often misunderstood the terms automatic and automated. Allocated time is not sufficient as the development team pushes the patching window or crosses the deadlines.
This time an automation test engineer thinks to find out all the reusable components so that the development time becomes less. We use a lot of loops to do the same activity again and again.
We try to automate whatever is possible. Our objective is to check an outline to check the overall completeness. Smart enough to complete the task???
standard become important as more people will be developing in a very short span of time. Getting feedback from the framework users will help for later acceptance.
Automation Test development is a dedicated effort, there is nothing called free lunch. (often management asks for free or extra time of a tester to automate the flow.) Yes, we can automate in a shorter span of time. but shall we??
No!!. we must not automate poor tests. Efforts put into automating high value, high quality tests will improve the test execution and provides greater payback. Similarly, simply automating manual test cases may not give us better ROI or the best automation suite. Select the tests judiciously.
Don’t try to automate very complex flows at first. Automation should go at a slow but steady pace. Identify the proper test case while automating the complex flows. If you automate chaos, you will get faster chaos. Smoke test cases are designed to check the health of the builds, releases. This type of test saves hundreds of hours for testers and developers. Smoke test cases are a good place to start the automation development, as they run almost every day, on every check in, or on every build. This is a less value add for testers error-prone and boring.
Extended environment and multitool Support:
Most of the designed Automation is not environmentally neutral. They support only one Operating system or one tool-specific.
Please build a framework that will work for multiple OS or in multiple tools. This can be achieved through designing a loosely coupled system.
The second most important issue is the data issue. Most of the time either data is not present or wrong data causes the execution to stop. Even using manual test data sometimes gives the wrong output.
The third point may be the data-hell condition. When we create a lot of modular scripts and try to create data for the modular approach we create the same data again and again.
Say I need to change a set of data there is no mechanism to make this latent change reflect everywhere. We have to open the child script datasheet one by one and change. there is no consistency at all between the data. These causes problem when running the scripts.
What I mean that any test can be operated from any machine. Let us improve the framework a little bit more. I can copy the master driver which will automatically create the folder structure add up the necessary dependent files and will be ready to run any test cases.
An Automation process is very much useful when a non-technical person can create an automation test case and execute the same. The basic building block of such kind of framework is to create business process. A non-technical person might not understand your piece of code but very well understand the Business process. He should not bother about the underlying code.
Ease of Functionality can be achieved when we can give the user a platform where the user can plug in and plug out any extra feature he/she required/not required respectively. Managers, business analysts, and functional people can control global testing without knowing the tool.
This will truly become Skill Utilization too. The more we make our framework user friendly, by providing Easy controlled and controllable actions, the More it will be a go-to tool for all. This feature will allow the business users to scope a set of test cases or scenarios for a specific business flow.
Easy Reporting and Self-documentation:
This is one more improvement that can be implemented is the self-documentation of steps performed. Gradually this feature will give proper navigation step validation and data validation in the future.
The new framework should support video recording of each and every test case execution report. Based on the requirements, the framework should support the passing video or fail video along with steps and logs.
Try less coding and try to reuse most of the possible combinations. This will reduce ambiguity. Maintenance is much easier when you have fewer lines of code. Redundant lines suck a lot in a later phase.
Most of the time after successful completion of development, our scripts got stuck. This is the time when we have very less no of time to complete the task. 80% of people say that the development of the scripts is not proper.
Even I found an instance where the “for” loop sucks if it got stuck in between of the loop. Understanding the loop also a big factor for the execution master who is in charge of execution.
sometimes we think if we can write statements instead of a loop. Basically, we look for consistency of test scripts at this point. We need to structure our automation regression test for the most successful execution. The main objective of the regression suite to give confidence and assurance, not detecting bugs.
Standards can be of different types:
- Coding standards
- Naming standards
- HIPPA or SOX guideline standards
- Folder Structure standards
- Library Architecture
- Easy plugin adapters
- Easy interface for framework
- Distributed Nature standards
Please use this whenever possible. Try to make a practice. This will reduce ambiguity inside the code. It is easy to understand when we have a proper Naming convention or we follow a proper guideline.. Also, we can use proper commenting to get a consistent test script. Framework should support/Accept only the consistent script.
When we are preparing a data-driven framework or testing a script for multiple data set, we require a lot more data files to be processed.
The framework must support plugins for data for driving the script or data pool to supply during testing of the same scenario for multiple data.
When a test engineer wants the same test to be executed on different data. The framework should support him with easy excel plugins of data and also give the flexibility to repeat the test how many times needed with different data.
This is continuous from the above point. How a nontechnical person will interact with our automation code? They need an interface to design the test.
The quality center is a classic example of an interface. But that is costly. The framework should have an interface where functional people can use that and prepare their tests.
The framework should have the flexibility to design and execute the test in different machines without any problem. Design and execution should be in Distributed mode.
How to put automation Standards?
The automation process starts with capture and playback. It is the beginning. We need to upgrade our script with validations, Synchronization points, conditional points that need to be added. We have upgraded from mouse-based co-ordinate testing to proper element testing. Slowly our codebase increases.
Apart from our regular test engineers, automation will be used by several other engineers(even functional test engineers too). It is important to have some standards in place.
More and more standards, if placed, automation will be streamed lined and this will reduce human dependency and human error. This will ensure the same meaning for all. Reduces the code duplicity.
This means the reduce the effort of coding, higher ROI. A structure if well documented and standardized yields more results and quality scripts than a quick undocumented and dirty approach. it will help to scale up the automation. An expensive tool is not necessarily needed or the best tool needs not to be picked. With the available tools, if we are able to put little standards, brings wonders.
Another important aspect of automation introduction in agile to provide coverage so that the development team has a safety net around them to change the code. Regression testing is only successful if we have such coverage and safety features around the development.
Continuous integration using jenkins etc and test a build-in automatic way will ensure the hygiene of development code. One can opt for any opensource tool to automate the testing with supporting infrastructure.
Quality of automation testing needs to be established.
Automation should have many folds. The major aspect again smokes or monkey way of testing the new build. Even the basic security, sanity, and deep drive into the application may come in handy in changing environments.
By putting the framework in place we can get quality scripts, reusable code, stability, and extendability. Putting standards, guidelines automation will be matured enough to go in unattended mode. Else management needs to incur the monitoring cost.
The standards can be divided into three sections.
- Development standards
- Cosmetic Section
- Infrastructure Section
- Naming conventions for variables –camel casing.
- The naming convention for global variable-must start with g
- The naming convention for the function name, class name follow starting of word with capital. also, the function needs to start with the return type.
- Documentation and open discussion-A great discussion often lead to great findings. Follow the principle -If you don’t know ask!!
- A how-to guide to writing a test script
- Standards for naming an object
- Data file construction
- Constant should be all upper case
The cosmetic section mainly talks about spacing. proper loop structure spacing. It is said code must be human-readable first and then machine-readable. The header is an important aspect of the cosmetic section.
Each function or class should have a header section to talk about the same. A comment must be included to make the code human-understandable.
here is a set of does not don’ts
- use of tagging gives better search facility. tagging can be mentioned in the commented area.
- only one statement per line
- maximum of 80 to 100 characters per line so that testers do not need to scroll right to see the content.
- indent sub-items’’
- use of proper parenthesis of using operators.
- turn on option explicit statement this will force variables to have certain values instead of junk, null values
- parameters need to be defined as pass by value or pass by reference
- keywords as if-then, etc should be in the same line.
Local- Use camel casing-small letter “l” in at the first letter of the variable.Like the first name. It denotes that it is a local variable.
Global-Same, if the scope is defined by global just use a small “g” at the beginning of the variable.Like-customer number it denotes that the variable is global and accessed from anywhere.
CONST- Use all capital words for variables that are constant in nature.Like-CONST CDATE.
Functions should start with FN_ followed by function name like- FN_getDate()
Similarly for Action is QTP should be named as AN_Action name like-AN_CreateCustomer
Append a small r before function and action if they are reusable in nature.
rFN_getDate() or rAN_CreateCustomer
The script name can be varied from framework to framework. The best practice is to give a module name before the script name. Like-CO_CreateCustomer(Co stands for Customer operation). It must be capital letters.
Function Library should start with FN followed by function library name and ends with a lib.like-FN_CreateOrder_lib
For a small project, we may go with .mtr but when the project goes big it is better to use the .tsr object repository.ORs should start with the Module name like-CO_Customer.tsr
a structured approach is always needed to work properly in a team together. It is important to structure the test assets(name,test data file,scripts,test cases,config file etc). Another best option to work with multiple environments is to use relative folder options.
The second approach is to share the function lists created by different automation. functions should be as generic as possible.
The best way to achieve this is to use
- One global function library
- Use business-specific library
Store all the structures in the version control system. Extra folders could be-
- Documentation-Related to framework
- General-Function to initialize the framework
- Language-Functions specific to any language
Advanced features/Recovery Scenario:
Any technical change should be caught by the framework itself and handle properly. In this way also try to capture the delayed fuse bugs like -memory leak wild pointers etc. While advancing forward, the framework should have the ability to handle unexpected exceptions. Every tool has its own way of handling exceptions. Use that to configure the recovery scenario. Please handle known issues with the code itself. As the more intelligence, you will put into a framework..it will slow down the process. It is kind of attaching a recovery Scenario to the test.
Ease of debugging, logging and feedback system:
Automation should give an easy debugging feature so that the code executed by any test engineers or a customer can debug the code easily. One of the best methods could be linear code generation.
Always listen or give proper value to the feedback of the users. They may pinpoint a lot of relevant process and practice issues. Even they may advise standards. This is very important for long term success.
Look for better ROI:
Easy Refactoring and update:
The maintenance process of a framework should be defined previously. The process to update a script /asset/code has to have a prior approval process and version control mechanism. Refactoring is an essential part of automation testing in order to restructure test code or test data.
- Increase internal efficiency
- Support upgraded technology
- Support change in functionality
- Increase maintainability.
When the script is old and running for say 3/4 years. It is extremely difficult without a proper comment or explanation.
At this time we try to understand from code itself what it is doing. Remember a more number of lines of code is costly during maintenance.
In agile, we often automate some flows which are unstable. Actually, there is no value add while automating unstable functionalities as maintenance overhead is huge.
This is the basic building block for an automation framework. Please visit Part-2 for the more advanced features. Better communication between testers and automation engineers is necessary to ensure framework design meets the testing objectives.
Any test framework design effort is a very serious factor for project development. Good planning, scheduling, and budgeting are required. It is important to get feedback from the testing team.
This may lead to greater learning and idea generation. there may be many cycles before perfection. Accept the shortfall and work towards it. If the project is complex, big and we are on a tight schedule. We need to scale back. Communication is the key here. While designing, we need to make sure-
- Find relevant information, code, examples from the internet. It is available in different forms.
- The automation may be developed by a group and maintained by another group. Transparency is the key.
- While designing the framework, we need to take care of business issues, people and process issues, and technical issues.