QA metrics that support decisions—and those that mislead without context
QA metrics should help a team decide where it is losing time and where users face the greatest risk. A test count, defect count or pass rate cannot show this on its own. A useful view therefore combines production outcomes, feedback speed and the health of the test suite.
Why a green dashboard can hide a problem
A team can have thousands of automated tests and a 99% pass rate, yet the single failing test may cover payment or sign-in. Another team may report many defects even though most have little impact and a critical defect escaped into production. The numbers are not wrong; without context, they answer a different question from the one the business needs to resolve.
The same problem appears when a metric becomes the goal. When a team is assessed by the number of tests it writes, simple checks naturally multiply. When it must reach a prescribed code-coverage percentage, it can add tests that execute lines without verifying important behaviour. The metric then improves faster than confidence in the release.
Start with the outcome for users
First, observe what happens after deployment. These measures show whether testing and the wider delivery process protect important functions:
- Defects that escape into production: classify them by severity and affected function rather than reporting one total. A payment outage and a footer typo do not have the same significance.
- Change fail rate: the proportion of deployments that require an immediate fix, rollback or other intervention. DORA defines this metric and evaluates it alongside delivery speed.
- Failed deployment recovery time: measure the period from failure to service restoration. A shorter time does not necessarily mean fewer defects, but it shows how quickly the team can recover.
- Failures of critical journeys: track purchase, sign-in, registration or another product-defining journey separately. One incident in a critical journey may matter more than dozens of minor findings.
DORA now uses five software-delivery metrics and groups them into throughput and instability. They do not replace QA metrics, but they give them business context: quality should not merely reduce defects; it should enable changes to be delivered safely.
Measure the time to a trustworthy result
A pipeline that turns green only after two hours provides late feedback. An average can hide slow runs, so it is useful to track both the median and a higher percentile such as p95: the time within which 95% of runs finish. Include queueing, environment preparation and test execution in the measurement.
Practical measures include:
- time from a code change to the first trustworthy result,
- duration of the pull-request check and of the full regression suite,
- time from a test failure to identifying its likely cause,
- proportion of runs repeated because of a test or environment problem.
These measures reveal whether feedback is being delayed by too many end-to-end tests, slow data setup or insufficient capacity. The test pyramid then becomes more than a diagram: it helps move suitable checks to a faster layer.
Track the health of the automated suite
A test suite can grow while becoming less trustworthy. It therefore needs operational measures of its own:
- Flaky rate: the proportion of tests or executions that pass and fail against unchanged code. Choose one definition and keep it consistent between reports.
- False alarms: failures caused by a test, its data or the environment rather than a product defect.
- Quarantined and skipped tests: report the number of disabled checks together with their age and owner. A skipped test must not appear as a successful one in the summary.
- Maintenance effort: compare the work spent repairing tests with the work spent creating useful new coverage.
- Diagnostic value: track how often the report is sufficient to identify the likely problem without another local run.
Instability needs history, not an impression from the latest execution. Our guide to measuring flaky tests describes a practical approach.
Numbers that distort the picture without context
The number of tests says nothing about the risks they cover. Ten precise checks of a critical calculation may be more valuable than hundreds of repetitive UI scenarios.
The test pass rate mixes critical and minor checks. Always pair it with failed critical journeys, skipped tests and known false alarms.
Code coverage shows which code a test executed, not whether it asserted the right outcome. The Google Testing Blog notes that a high percentage does not guarantee high-quality tests. Coverage is most useful for locating untested areas, not as standalone evidence that a release is ready.
Defects found per tester encourages people to split findings, compete for easy defects and penalises those who prevent problems. Quality is an outcome of product, engineering, operations and QA working together, not an individual scoring contest.
Automation percentage has no universal target. Some scenarios are cheaper and more informative when left to a person; what matters is whether automation shortens feedback and protects recurring risks. It helps to identify tests that are not worth automating.
A small dashboard is more useful than dozens of charts
Six views are usually enough to begin: critical production defects, change fail rate, failed deployment recovery time, p95 feedback time, flaky rate and long-skipped critical tests. For each one, agree on its definition, data source, owner and the decision it is supposed to trigger.
Compare the trend of one product over time. Ranking teams without accounting for different architectures, risks and release frequencies invites the wrong conclusions. A worsening metric should lead to a question and a specific intervention; if it merely glows on a screen, it adds no value.
What you gain
A well-chosen set of metrics shows whether risk to users is falling, whether the team receives feedback in time and whether automated tests remain trustworthy. It also helps distinguish a product problem from a test-suite or pipeline problem. The result is not a universal quality score, but evidence for deciding what to improve first.
Next step
Choose one product and review the last three to six months. List the decisions currently made by intuition alone, then assign one outcome measure and one process measure to each. If a measure has no clear definition or owner, repair the collection method before setting a target. A QA audit can add a broader view of the process and its blind spots.