How to measure flaky tests and identify instability patterns
When the same test sometimes fails and sometimes passes without an obvious change, the result is described as unstable or flaky. Simply retrying the test does not reveal whether the problem is in its code, the application, the data or the environment. Stabilisation therefore starts with measurement, not with longer timeouts.
Why one failed run is not enough
The same error message can have different causes:
- Application defect: the test correctly exposed incorrect behaviour that may depend on timing or concurrency.
- Test defect: the test did not wait for the right state, used a fragile selector or assumed an unsuitable order of events.
- Data or environment issue: an expected record was missing, a service responded slowly or the infrastructure lacked the necessary capacity.
If the test is merely retried and the first result is discarded, the team loses the evidence needed to distinguish between these possibilities. The number of retries increases over time while the cause remains unresolved.
What is worth recording
The foundation is a history of individual attempts, not only the final green or red pipeline status. For each test, track:
- the outcome of the first attempt and any retry;
- the run time and the point at which it failed;
- the error message and any recurring pattern;
- the application version, browser, test environment and number of concurrent workers;
- the account or test-data identifier used;
- available diagnostic evidence, such as a log, trace or screenshot.
Two practical metrics are the proportion of failed first attempts and the proportion of tests that pass only after a retry. The second reveals instability that a final green report may hide. There is no single suitable percentage for every project; prioritisation should also reflect whether a test protects a critical process.
How to carry out controlled measurement
1. Fix the starting conditions
Choose the same application version, a known configuration and controlled data. Run the suite several times without changing these conditions. A different outcome is a sign of instability, not automatic proof that the test itself is defective.
2. Change one condition at a time
Compare a test running alone with the complete suite running concurrently. If the problem occurs only under concurrency, tests may be sharing data or resources. Comparing faster and slower environments can expose unsuitable waiting logic. A difference between browsers may be a compatibility defect rather than a flaky test.
3. Group matching failures
Ten tests may fail because of one shared cause, such as unavailable authentication or a data-setup error. Grouping by error message and point of failure prevents the team from fixing every symptom separately.
4. Classify results by the next action
- repair the test or its data;
- investigate an application defect;
- stabilise the environment;
- temporarily separate the test from the blocking suite and set a repair deadline;
- remove, by agreement, a test that is obsolete or duplicates another check.
Temporary separation must not allow a test to disappear without an owner. It should remain visible in reporting together with the reason and the date of the next decision.
What to watch out for
Increasing a timeout can be justified when the application genuinely needs longer for an operation. It is not a universal fix. If the test waits for the wrong signal or uses shared data, a longer timeout merely delays the failure and extends every run.
Deletion also requires care. A test may look outdated while still representing an important edge case. Confirm its purpose with someone who understands the product before removing it.
What you gain
- A specific list of unstable tests instead of a general impression.
- Causes separated between the test, application, data and environment.
- A repair order based on frequency and user impact.
- Metrics that can confirm whether a repair produced an improvement.
Next step
Start with first-attempt results from recent releases and identify tests that passed only after a retry. If you do not have this history, contact us. We can design a short, controlled measurement and classify the suite into tests to repair, investigate or retire. If an unsupported tool is the main obstacle, test migration may be more appropriate.