Fix test suite

How to manage a flaky test quarantine

A flaky test can block a change even though it passes on a retry without any code modification. Simply disabling it leaves the team without control of the risk or evidence for a repair. Safe quarantine keeps the test visible in a temporary, non-blocking process with an owner and agreed return conditions.

Quarantine is not a folder for every red test

A test belongs in quarantine when there is evidence of instability under comparable conditions. For example, the first attempt failed and a retry passed against the same application version, environment and expected result. One failed run is not enough. The cause may lie in the product, test, data or infrastructure.

A consistent application defect is not flaky merely because it blocks a release. If a payment records the wrong amount every time with the same inputs, the test should remain red. The team must fix the product defect or consciously accept the release risk. Quarantine would only bypass a valid check.

An intermittent application defect also needs care. A variable result does not prove that the test is faulty. Concurrent requests, delayed processing or time-dependent behaviour may expose a real product problem. Before admission, classify the test using the process for measuring flaky tests. The root cause can remain unknown, but you must know which risk will temporarily stop blocking the release.

Set admission rules

Do not let several pipeline retries and fatigue decide admission. Agree short rules and use them for every candidate. The record should answer at least these questions:

• which test and configuration are unstable,
• where to find the first failure and subsequent attempts,
• what the test verifies and which product risk will temporarily stop blocking,
• who approved the move and who owns the repair,
• when the team will review the decision,
• what will allow the test to return or be removed.

If the first failure is missing, run the candidate under controlled conditions and collect evidence before moving it. Do not create a quarantine entry without an owner or review date. Such an exclusion has no end.

The owner need not repair the code personally, but remains accountable for the next decision and a response. This split follows the wider question of who owns automated tests.

The review date is not a promise that the repair will be complete by then. It is the point when the owner must confirm the next action. They may return the test, extend quarantine with a new reason, replace it with another check, or propose its removal. Deletion has separate criteria, described in which automated tests to delete.

Separate only the smallest necessary scope

If a scenario fails only in Firefox, do not exclude its variants for other browsers. If one combination of role and data causes the problem, quarantine should not remove the whole file or module. The wider the disabled scope, the more valid checks lose their influence over a release.

The same rule applies to a shared cause. Ten failing tests may conceal one broken data preparation step. Record them under a common incident, but label each test so it can be assessed separately after the repair. Disabling a whole directory makes return harder and misrepresents the amount of uncovered risk.

Consider a checkout scenario that fails intermittently only in Firefox when it uses a discount account. Quarantine that combination, not the whole checkout journey or every browser. The ordinary checkout check can continue to block changes. If the same assertion later starts failing consistently in the other browsers, classify the incident again. A quarantine label must not prevent the team from recognising a new and wider product problem.

Preserve the first failure and its context

Retries are useful, but they must not overwrite the first result. With retries enabled, Playwright classifies tests as passed on the first attempt, flaky when they pass only after a retry, or failed when all retries fail. Failed tests are not retried by default. The official Playwright retries documentation describes this behaviour.

Attach the commit identifier, application version, environment, browser, configuration, failed step and error message to the quarantine record. Retain any available trace, log, screenshot or network data in line with the project’s sensitivity and retention rules. The outcome of every retry matters, not only the final green mark.

Check the evidence before moving the test. A retry may clear data, restart a worker or use a different account. The failure may disappear precisely because the conditions changed. That result helps diagnosis, but it does not prove that the original state was healthy. Reporting should expose the change rather than folding all attempts into one outcome.

Keep the test running in a separate non-blocking path

Quarantine should separate an untrustworthy result from the change decision, not stop observation. A practical setup has two runs. The main CI run excludes labelled tests and remains blocking. A separate quarantine run executes them against the same commit in a comparable environment, but its result does not prevent a merge or deployment.

In Playwright, a custom tag and filtering can provide this split. The documentation explains that tags can be selected with --grep and excluded with --grep-invert. A custom annotation can carry a type and description, such as an issue link, and is available to reporters. See the Playwright annotations documentation for details. Neither a tag nor an annotation assigns an owner, sets a review date or records accepted risk. The team process must do that.

The separate run needs a regular schedule and a result notification. If it runs only during a manual investigation, nobody can see whether the behaviour improved, deteriorated or stopped executing altogether between reviews.

skip and fixme are not a quarantine process

Playwright’s test.skip() marks a test as irrelevant and does not run it. test.fixme() also prevents execution, and the documentation recommends it when running the test is slow or causes a crash. Both annotations can be useful for a particular configuration, but neither creates a quarantine. They do not create a separate run, assign an owner or define return conditions.

Skipping is appropriate when a test does not apply to a configuration, perhaps because a feature is unavailable in a certain product variant. A flaky test still applies. Labelling it only with skip or fixme stops the team collecting the results needed for a repair. A test that damages the environment or prevents the rest of the run may be an exception. Even then, it needs a record, an owner and another way to verify the risk until it can run safely.

Report skipped results separately from passes

A green report must not count a skipped test as successful evidence. At a minimum, distinguish tests that passed on their first attempt, flaky results after a retry, failed tests, skipped tests and quarantine-run outcomes. For the quarantine itself, track the number of open entries, their age, overdue review dates and risks without a replacement check.

No numerical threshold for admission or return suits every project. One intermittently failing test may protect payments, while another checks cosmetic sorting. Repair priority should reflect the impact of the protected scenario, first-failure frequency, investigation cost and the amount of feedback the test obscures. Address entries that create the largest coverage gap or consume the most assessment time first.

Agree return conditions in advance

A test should not return merely because somebody ran it locally once and it passed. Before return, the cause should be understood or the team should at least verify a change that removed the observed failure mechanism. The change goes through code review, and the test runs repeatedly in the configurations where the problem appeared. The team chooses the run count and observation period according to risk and history, not a universal percentage.

At exit, confirm that the test passes on its first attempt rather than only after a retry. Remove the quarantine label, restore the scenario to the blocking run and verify its result in CI. Only then close the task with links to the repair and verification results. If the scenario no longer has value, this is not a successful return. It is a documented decision to replace or remove the test.

What a managed quarantine changes

A release is no longer blocked by a result the team cannot yet trust, but the problem remains visible. Preserving the first failure shortens later diagnosis, the separate run shows how behaviour changes, and an owner with a review date prevents a temporary exclusion from becoming permanent.

Start by listing skipped tests and results that passed only after a retry. Add the risk, evidence, owner and next review date to each. If you need to set up quarantine or return long-disabled tests to CI, contact us.