I have been toying with Exam Automation for the earlier 15 several years and a person factor that keeps coming up time and again is how to evaluate the achievement of automation.
To begin with, we utilised to measure% of examination instances that were automatic. We set a purpose – X% of exam conditions that desires to be automated within a time interval and then we made use of to go about measuring the progress, pattern and sense good when we defeat the goal.
We shortly figured out that this has major limitations. For case in point, we could have 70% exam scenarios that have been automatic, but throughout a particular regression cycle, we may execute only 25% – due to the fact others have been not related for that regression cycle. Pretty shortly, we ended up with some fascinating (read through 'heated') discussions with stakeholders on how can 70% fall to 25% and so on. automation engineers utilised to complain that the stalkers did not comprehend what was associated with take a look at automation and regression screening. So, in the finish – however we nor the quakeholders had been happy with the condition.
It does not have to be this way. Let us go back again to the essentials.
Why do we do Exam Automation? To enhance examination performance.
Ok – so, what does that necessarily mean? It signifies that we can conserve exertion, time and potentially charge as properly.
Alright – if that is what the worth we are meant to get from take a look at automation – they why are we NOT measuring that?
As an alternative of measuring% of take a look at circumstances that had been automatic, take a look at automation metrics should really emphasis on the cost savings we get from executing individuals exam cases – in conditions of hard work and time?
It seems to be simple. But calculating the exertion and time cost savings as a result of test automation execution is something but simple. For arriving at the savings –
- We need to know how considerably time and work it takes to execute every handbook examination situation
- We want seize that someplace
- We have to have to then map this to the real exam automation scripts that had been run and then calculate this.
This will require some hard work, collaboration with the handbook take a look at staff to get there at this.
Agreed. But then the% take a look at scenarios that were being automatic was not bringing out the value that examination automation was supposedly to obtain, so it would make feeling to improve it to Take a look at Automation Worth metric.
We did just that and started measuring the test automation price savings. Trust me, it was distressing to gather this details – the test management resource that we were being applying did not aid guidance capturing the info, aggregating it and offering us the metric along with the pattern. We experienced to depend on our good previous pal MS Excel to get there at this. At times the savings are obtained above many check cycles.
But the outcomes have been well worth the agony. As soon as, we shifted the metric to automation discounts, we shifted the behavior in the proper path. Due to the fact we estimate the effort and time cost savings on what we execute, examination automation engineers intensely targeted on automating the exam situations –
- That was energy intense, which gave the most effective price savings
- That ended up company critical, due to the fact substantial precedence test scenarios get executed extra frequently through regression cycle
- Take a look at situations that had to be executed across different configurations – OS, Browser, DB, devicecombinations – because they resolved in large cost savings.
Folks are likely to align their behaviors based mostly on what will get measured. When we evaluate the correct items, we permit the suitable behaviors. Make sure you share what are the metrics that made much better business enterprise feeling and how it adjusted the behavior in the correct way. Many thanks