Testing Metrics – What Gets Measured, Gets Done!

“What gets measured gets done. What gets measured and fed back gets done well. What gets rewarded gets repeated.”

The above stated pithy statement relates very well to the need and importance of testing metrics. When an organization is able to clearly and explicitly define the testing metrics it requires, and then is properly able to analyze them and use the analysis to fix the existing issues, it will invariably be treading on the right path towards success and growth.

In normal parlance, most organizations follow the well tread path of plan, do, check, act (- better known as PDCA) when beginning any new venture or project.

As per WhatIs.com, “PDCA (plan-do-check-act, sometimes seen as plan-do-check-adjust) is a repetitive four-stage model for continuous improvement (CI) in business process management. The PDCA model is also known as the Deming circle/cycle/wheel, Shewhart cycle, control circle/cycle, or plan–do–study–act (PDSA).”

Plan: In terms of a product/software development/testing lifecycle, Planning refers to defining and laying out specific Business Goals and gaining a thorough understanding of the need for the planned application. At a later stage, this also includes testing the product, collecting statistical data, identifying and ascertaining the root causes of the issues being faced, and planning for fixing them.

Do: This is the stage where organizations define and decide upon the multiple measurement variables and metrics. These metrics will help understand the effectiveness of the product as also help measure the quality of the product. This stage also involves developing and implementing solutions for the identified issues.

Check: This stage is used to analyse the reasons as to why a product is behaving in the way it is, as also to compare the data before-and-after a fix has been made. This stage requires you to document the observations, inform the team about any changes to the process, and also recommend changes that need to be made.

Act: As the name implies, this stage involves taking Corrective Actions and fixing the product to come up with a quality product.

As seen above, one of the most important link between all the 4 activities – PDCA – is metrics and its measurement. The question to ask is – what should we measure, and when should we start measuring?

What sort of metrics need to be collected and analysed must be decided in the planning phase. A few metrics that matter – especially when testing a product for quality – are as follows:

• User Story Coverage
• Planned vs Done Story Cards
• Test Automation Coverage
• Automated Test Effectiveness
• Mean Time Between Failures (MTBF)
• Mean Time To Repair (MTTR)
• Overall Equipment Effectiveness (OEE)
• Defect Rejection Rate
• Production Defect Leakage
• Defect Severity Index
• Defects by Sprint
• Deployment Lead Time

Unless we specify what needs to be measured, we will never really be able to give due attention and focus to completing it to the best of our abilities. Measuring the correct variable, analysing it correctly, and then performing the required tests will lead to creation of better products and make them more reliable and robust.

“Not everything that counts can be counted, and not everything that can be counted counts.” ~ Albert Einstein

That said, what is even more important than measuring metrics is the follow up – what is the actual improvement we have made in the product based on our findings. Identify your top-most, business-critical priorities, analyse the metrics against them, and then make the fixes required.

Zooming in to the top-most priorities and then working against the related metrics will help an organization provide better quality products, reduced go-to-market time and hence improve their ROI.

Experts at Cigniti can help you understand what needs to be measured and tested to get the optimum outputs. Get in touch with us for meeting all your testing requirements.