I have been toying with Examination Automation for the earlier 15 a long time and just one detail that keeps coming up time and all over again is how to evaluate the achievements of automation.
At first, we employed to evaluate % of test scenarios that had been automated. We established a goal – X% of test instances that desires to be automatic in a time interval and then we employed to go about measuring the progress, pattern and feel good when we defeat the goal.
We before long figured out that this has really serious limitations. For illustration, we may have 70% examination circumstances that were automatic, but in the course of a distinct regression cycle, we could possibly execute only 25% – simply because others were not pertinent for that regression cycle. Really shortly, we finished up with some fascinating (browse ‘heated’) discussions with stakeholders on how can 70% drop to 25% and so on. automation engineers utilized to complain that the stakeholders did not fully grasp what is included with test automation and regression testing. So, in the conclude – neither we nor the stakeholders had been delighted with the predicament.
It does not have to be this way. Allow us go again to the basic principles.
Why do we do Exam Automation?To increase check performance.
Ok – so, what does that necessarily mean?It suggests that we can help save effort and hard work, time and possibly value as perfectly.
Alright – if that is what is the worth we are supposed to get from check automation – they why are we NOT measuring that?
As a substitute of measuring % of test conditions that had been automatic, exam automation metrics should really emphasis on the personal savings we get from executing people exam conditions – in terms of work and time?
It looks basic. But calculating the effort and hard work and time financial savings via examination automation execution is anything but very simple. For arriving at the financial savings –
- We need to have to know how substantially time and effort and hard work it can take to execute each individual handbook take a look at circumstance
- We will need capture that somewhere
- We need to have to then map this to the true exam automation scripts that have been run and then determine this.
This will need to have some effort, collaboration with the guide exam team to arrive at this.
Agreed. But then the % examination conditions that were automatic was not bringing out the price that exam automation was supposed to attain, so it tends to make perception to modify it to Exam Automation Price metric.
We did just that and commenced measuring the check automation financial savings. Have confidence in me, it was agonizing to gather this information and facts – the take a look at administration device that we were being applying did not readily assistance capturing the info, aggregating it and supplying us the metric together with the pattern. We had to rely on our fantastic aged close friend MS Excel to arrive at this. At times the savings are achieved above several examination cycles.
But the effects have been well worth the ache. At the time, we shifted the metric to automation personal savings, we shifted the behaviour in the suitable route.Because we estimate the exertion and time cost savings on what we execute, take a look at automation engineers seriously targeted on automating the take a look at circumstances –
- That were effort intense, which gave the greatest savings
- That were being small business important, given that high priority take a look at circumstances get executed far more normally for the duration of regression cycle
- Examination conditions that had to be executed throughout diverse configurations – OS, Browser, DB, devicecombinations – due to the fact they resulted in substantial discounts.
Individuals tend to align their behaviours based on what will get measured. When we evaluate the suitable points, we empower the suitable behaviours. Make sure you share what are the metrics that built much better business enterprise feeling and how it modified the behaviour in the ideal route. Thanks