Follow @techarcis

Products and new features roll out in competitive market demands shortened quality testing cycles and time to market. With today’s new age applications, performing tests manually like regression testing and running the test cases multiple times can be laborious and time consuming. And one cannot compromise on the quality of the end product at the same time. So, automation in testing arena plays a vital role and become inevitable in IT industry. By way of automation, these test cases take lesser time to run the regression testing cycles.

Automaton regression testing not only results in saving time but also costs, especially if the test cases don’t change frequently and regular test cycles are required. Regular maintenance activities like patches or upgrades over the application’s lifetime can result in working features to regress; meaning regression cycles can be run for any change which happens in the environment. But having said all that, automated testing cannot be beneficial at all the times. We need to analyse the situation in terms of ROI and see where the automated testing is truly benefiting us.

Metrics and Measures of ROI

The value of test automation can be truly analysed by considering the metrics and measures which can help the project team members, managers as well as the stakeholders. ROI on test automation can be known by many ways but we will first consider the metrics like test coverage, time gain and defects found. Good test automation metrics can be easily related to main project attributed like quality, cost, risk and schedule. Below is the outline of few such metrics for test automation.

Coverage for test automation

Test automation coverage metrics signifies the number of test cases automated vs. total number of test cases that are automatable. Improved test coverage signifies finding the defects faster, enabling teams to fix then earlier, thus leading to high quality and lower risk.

Time gained for every regressing cycle

With agile and devops everywhere, there are daily and weekly builds in a product lifecycle, wherein the need to run regression cycles frequently or sometimes even daily becomes obvious. Automating these tests can cut the testing time significantly, enabling the teams to run them multiple times in shorter time duration in turn reducing the overall project schedule and cost.

There is cost saving with automation but one should not forget the initial cost of automating the test cases i.e. the efforts and tools needed to automate which is always high but with every automated test cycle, there is some cost saving and after a number of test cycles, the cost saving for each test cycle will add up to the initial cost of automation.. The project team is required to detect this inflection point depending upon the number of test cycles.

Defects detected per regression cycle

The number of defects found with every regression cycle indicates about the quality of the product and effectiveness of the automation procedure.

The True ROI for test automation

The ROI for test automation can move into positive zone depending upon the number of regression cycles required or projected for the lifetime of the product or application.

The classic ROI for test automation can be simply calculated by the below formula:

ROI= (Cost of manual testing – Cost of automation)/Cost of automation

This formula looks right, isn’t it? But the problem with classic ROI calculation is, we cannot compare automation testing with manual testing as executing the same number of manual tests as automated ones is near to impossible. The true Automation ROI value is the benefit of this type of testing and it can be: a) Reduced time to market, b)Increased test efficiency (Productivity),c) Increased test effectiveness.

Teams also need to consider the consistent cost for automation, primarily related to maintenance. Every application requires changes throughout its life, all these changes effect test automation scripts that need to be changed depending on the change in the application.

Automation does produce tangible and intangible benefits. Tangible benefits might be related to hours saved and time gained, intangible benefits can comprise faster feedback from people, identifying defects faster in the development cycle.

More Related Blog