How AI can triage hundreds of test failures without hiding a defect
After a large release, one hundred tests may fail at once even though there are only three causes: an unavailable test environment, a change to one shared element, and a genuine application defect. A language model can help group similar outputs and prepare hypotheses. It must not decide without evidence that a test is flaky, nor automatically rewrite a test merely to make the result green again.
Why the number of red tests is misleading
One defect can trigger a chain of failures. If sign-in is broken, order, profile, and administration scenarios may all fail. Conversely, two failures with the same Timeout message need not share a cause: one test may have waited for a slow API while another looked for an element that no longer exists on the page.
Opening every report manually is slow and encourages shortcuts. A team may rerun the entire job repeatedly, label tests as unstable in bulk, or increase a timeout. These actions merely postpone the underlying cause. A controlled collect → redact → cluster → hypothesis → human verification flow is more useful: gather evidence, remove sensitive information, create groups, formulate hypotheses, and have a person verify them against the original evidence.
A realistic example: 126 failures after one change
After a checkout change is deployed, an overnight run reports 126 failed tests. Sorting by the final error message creates a large Timeout group, but this group mixes several problems. After artefacts are normalised, AI initially proposes two broad groups and leaves thirteen cases unassigned. Human review then divides the result into three confirmed groups.
The first group contains 83 scenarios that stopped during sign-in. They share the same authentication-service response and began failing at the same time, so a person checks service availability and one representative test. The second group contains 21 tests with missing data after a failed setup job. Repairing the environment removes both groups without changing the tests.
The third group contains nine checkout scenarios. AI initially places them with the general timeouts, but each has a successful network response and the trace shows an incorrect order total. A reviewer separates them as a possible application defect. Reproduction with the relevant line items confirms an incorrect discount. If the group had been automatically labelled as an environment problem and merely rerun, the real regression would have remained hidden.
The remaining thirteen failures do not have enough shared evidence. They stay in an “undetermined” category and receive individual review. This outcome is more useful than forcing 100% of cases into groups: the team resolves two bulk causes without losing a smaller group that has business impact.
1. Collect: gather comparable evidence
For each failed test, retain at least its stable ID, build and commit, time, environment, browser or device, error message, stack trace, and last relevant steps. Depending on the tool, add network and application logs, a screenshot, or a trace. Playwright Trace Viewer, for example, lets an investigator review an action timeline, DOM snapshots, console output, and network requests.
Keep the result history as well. A single failure does not show whether the problem is a reproducible regression, random instability, or a one-off environment outage. We discuss practical measurement in How to measure flaky tests and identify instability patterns.
The evidence should follow a consistent structure. If one test provides a complete trace while another provides only “element not found”, the model will form groups based partly on input quality rather than solely on cause.
2. Redact: remove sensitive information before AI sees it
Logs and traces may contain access tokens, cookies, personal information, form contents, request headers, or entire API responses. Redaction must therefore happen before data is sent to a model, not during a later review of the model’s answer.
Automated rules can replace tokens, email addresses, and identifiers with consistent placeholders. Consistency matters: <USER_17> across several files still allows events for one test account to be connected without disclosing the original value. Test the rules on real artefacts because a secret can also appear in a URL, screenshot, or attachment. Model choice, processing location, and retention rules depend on the specific solution; Where does your data go during AI-assisted testing? covers the broader context.
3. Cluster: group failures using several signals
First create a deterministic foundation: normalise stack traces, remove changing timestamps and IDs, and calculate a fingerprint from the error type, code location, and last step. AI can then add semantic similarity across messages, logs, and visible symptoms.
The output should contain more than a group name. For each cluster, retain:
- the included tests and runs,
- shared signals that support the grouping,
- differences and contrary evidence,
- a representative example,
- uncertainty and items that could not be assigned.
Forcing every failure into a group is dangerous. An “unknown” category is better than a convincing but unsupported merger of a real defect with an environment outage.
The output must be reviewable, not merely convincing
Each group needs a stable identifier and concise name, but also enough information to evaluate the decision later. A cluster record can include the dominant error fingerprint, affected tests and builds, first and last occurrence, shared artefacts, exceptions, and the proposed next diagnostic step. A link to the original trace is more valuable than a long summary without a source.
Keep three values separate: similarity, cause hypothesis, and priority. Two tests can be highly similar while their cause remains unknown. Conversely, one failed payment scenario can have higher priority than fifty failures in an internal report. A model may propose all three values, but they should not be collapsed into one ambiguous “confidence” score.
Retain the version of the normalisation rules, prompt, and model configuration as well. The goal is not to reproduce every word of an answer, but to explain why results changed between two runs. If a new version starts creating broader groups, the team has evidence for comparison rather than an impression that AI has simply become more confident.
4. Hypothesis: formulate an explanation, not a verdict
For one cluster, a model might propose: “The tests failed after a shared sign-in button locator changed.” It must attach supporting evidence, such as the same commit, a DOM snapshot without the original element, and a consistent failure location. It should also name alternatives: a partially loaded page could produce the same symptom.
A hypothesis must not directly modify the repository, increase a timeout, disable a test, or create a production incident without the agreed approval. Applying the flaky label requires particular care. It needs a history of repeated runs under comparable conditions, not merely similarity to a previous unstable test.
5. Human verification: a person confirms cause and impact
A tester or developer selects a representative failure from each cluster and reviews the original artefacts. They inspect the application change, reproduce the scenario or rerun it under controlled conditions, and compare the result with the hypothesis. For a large cluster, they also inspect several edge members so that two causes are not hidden under one name.
Only then does the team assign a state: application defect, test defect, data problem, environment problem, or not yet determined. The fix is made in the appropriate layer, and a subsequent run confirms that it removed the cause without weakening the assertions. If the suite is broadly damaged or untrusted, triage findings become input to repairing the existing test suite, not an automated substitute for that work.
Decision boundaries AI must not bypass
Before adoption, define the operations that remain subject to human approval. AI may recommend another run, quarantine, or locator change, but it should not disable a test, alter an expected result, extend a timeout, or close a defect report by itself. It should likewise not write to production systems or assign an incident without separately approved rules.
Applying the flaky label requires observing alternating success and failure under comparable conditions and examining possible differences in data or environment. Even strong similarity to a historically flaky test is not proof. Temporary quarantine can protect a pipeline from noise, but it needs an owner, reason, review date, and retained failures; otherwise it becomes a permanent way to hide risk.
When artefact quality is poor, the system should request more evidence or leave the case undetermined. Without a stack trace, history, and environment information, saying “we do not know” is safer than inventing a specific cause from the test name.
How to verify that triage actually helps
Before regular use, prepare a small reference set of historical runs for which people know the confirmed causes. Hide those labels from AI, let it create groups, and then compare the result. Pay particular attention to incorrect mergers of distinct causes because they can hide a real problem. Producing too many small groups mostly reduces the time saving, but it is usually easier to notice.
In operation, measure the proportion of clusters confirmed by people, how many clusters are split or merged after review, the proportion of undetermined cases, and the time to the first confirmed cause. The number of failures processed automatically says nothing by itself about correctness. Compare the results with the previous manual process, and repeat the evaluation after a substantial change to the suite, reporter, or model.
A useful system gradually reduces repeated work without increasing incorrect mergers. If it creates quick summaries but a reviewer still has to inspect every artefact from the beginning, or regularly discovers hidden distinct causes, adjust the inputs, clustering rules, or AI scope rather than merely rephrasing the prompt.
This emphasis on evidence, evaluation, and human validation is consistent with the general generative-AI risk-management approach in the NIST AI RMF Generative AI Profile. NIST does not prescribe this particular five-step workflow; it is a practical design for test-failure triage.
What you gain
Instead of one hundred isolated messages, the team receives a smaller set of working groups supported by traceable evidence. It can find a common cause sooner while retaining visibility of ambiguous and distinct failures. A person still decides what the defect is and how to fix it, so time savings do not come at the expense of trust in the suite.
Next step
Introduce a consistent artefact format and redaction rules for one failed run. Let AI propose only clusters and hypotheses, then manually verify both a representative and an edge case from each group. Add the workflow to regular run evaluation only after it repeatedly distinguishes causes without hiding defects.