Fix test suite

What is regression testing and how often should you run it?

Regression testing checks whether a change has damaged behaviour that previously worked. It is neither one large “test everything” button nor a mandatory nightly run. It is a selection of checks at code, API, integration, and user-interface level whose scope and timing should reflect the risk of the particular change.

A regression is not only a defect in the changed feature

A change can have an adverse consequence in the modified component, a shared library, the database, environment configuration, or a connected system. A regression is precisely this kind of side effect: the behaviour worked before the change, but afterwards it no longer works or behaves differently even though that was not intended.

Regression testing therefore answers the question: What else could this change have damaged? Its scope starts with impact analysis, not a list of every test that exists. A change to price calculation may affect the basket, payment, invoice, accounting export, and refund. A database upgrade may make migrations, query performance, and compatibility with older records relevant.

The official ISTQB Foundation Level syllabus likewise distinguishes adverse consequences in the same component, other parts of the system, connected systems, and the environment. Regression tests therefore do not have to be end-to-end scenarios in a browser.

Regression testing and retesting have different objectives

Fixing a defect calls for two related but distinct checks:

• Retesting, more precisely confirmation testing, repeats the scenario that originally failed and confirms that the specific defect has been fixed. If a discount code deducted its value twice, the confirmation test uses the same conditions on the corrected version.
• Regression testing checks whether the fix caused a new problem elsewhere. It might cover a normal order without the code, a discount combined with delivery, the amount sent to the payment gateway, and the data passed to accounting.

A green confirmation test therefore does not mean a green regression run. It only confirms the correction of the known symptom. Conversely, a regression suite without the confirmation test may pass while the original defect remains because it did not include that exact failing case. A defect-fix plan should contain both questions and state the expected result for each.

How a fix can pass while the release still breaks

Imagine an online shop in which a ten-per-cent code was applied twice when combined with a particular loyalty discount. A developer modifies the shared service that calculates the total price. They repeat the original scenario, obtain the correct amount, and include the fix in the release.

The same service also prepares the payment request and order data. For an order containing a gift voucher, the payment gateway now receives the discounted amount while the stored order contains the amount before the final recalculation. Payment succeeds, but webhook processing rejects the amount mismatch during reconciliation and the paid order remains in “awaiting payment”. The discount-code confirmation test is still green; an unintended side effect has broken the release.

Proportionate regression testing would not have to click through every combination in the UI. Unit tests could cover the rules for combining discounts, an API or integration test could compare order and payment amounts, and one critical end-to-end (E2E) test could cover the complete purchase. The goal is to cover the path of the risk at the appropriate level, not maximise the number of browser steps.

Regression testing belongs at several levels

A suite made up only of UI tests is often slow and harder to diagnose when it fails. Distributing checks according to the test pyramid allows most problems to be found closer to their source.

• Unit and component tests check calculations, validation, state transitions, and the behaviour of small parts. They suit many combinations and boundary values.
• API and integration tests exercise business rules, the database, contracts, and communication between components or external services. This is often where most regression coverage for shared processes belongs.
• UI and end-to-end tests confirm that representative user journeys hold together from the interface to the outcome. They should protect critical connections, not repeat every lower-level combination.
• Manual and exploratory testing complements automation where the team needs to assess new behaviour, usability, or a risk that does not yet have a stable test.

The technology does not make a test a regression test; its objective does. A unit test of an existing rule and a manual check of an established process can both be part of regression testing if they look for possible side effects of a change.

Three layers instead of one endless suite

It is practical to divide regression coverage by feedback speed and breadth of risk:

1. The fast layer contains stable unit and component tests, selected API tests, and a few critical checks. It runs for a relevant change and should stop an obvious problem early. After deployment it can be complemented by a smoke test, whose purpose is explained in the comparison of smoke, sanity, and acceptance tests.
2. The wider layer adds integrations, more business rules, permissions, browsers, or representative E2E journeys. It runs after changes are combined, against a release candidate, or after an event that affects shared parts.
3. The full layer is the agreed maximum coverage relevant to the product, not automatically every test ever created. It is useful for a significant release, migration, infrastructure change, or other high-risk event. It may combine automated runs with targeted manual checks.

Each layer needs a target feedback time, an owner, trigger conditions, and an agreed response to failure. Without them, “fast” and “full” become folder names whose contents eventually stop matching their labels.

How often to run regression tests

There is no universal interval. A team that deploys several times a day needs a different arrangement from a regulated product with scheduled releases. A better question than “daily or weekly?” is “which event changed our risk?”

A scheduled overnight run can be useful for the slower part of the suite, but the calendar does not replace a trigger. If a critical change waits until night, feedback may arrive too late. If the product has not changed, another full run may consume capacity without providing new evidence for a decision.

How to select and prioritise tests

Start with an impact map: which components changed, what data they read and write, who calls them, and which user or operational processes depend on them. Then combine four signals:

• the impact of failure on customers, money, data, security, or operations;
• the likelihood that the change affects the journey;
• the area’s history of defects and instability;
• the strength of other evidence, such as code review, unit tests, or monitoring.

The result should not be a falsely precise score. A useful order is enough: tests with high impact and a direct connection to the change first, shared integrations and critical journeys next, then lower risks. The article on risk-based testing develops this process.

Selection based on changed files or tags can reduce the suite substantially, but it needs a safety net. A dependency map may miss dynamic configuration, shared data, or an indirect dependency between services. Targeted selection should therefore be combined with a small stable set of critical journeys and, where the risk warrants it, a wider run.

The suite must evolve with the product

A regression suite is not a finished inventory. A new feature creates new risks, removed behaviour makes some tests obsolete, and moving a rule between services may change the most appropriate test level. For every significant change, update not only test code but also the risk map, tags, ownership, and layer assignment.

A defect that escapes to production prompts a gap analysis, not an automatic instruction to add another E2E test. First determine why the existing checks missed it. The missing element might have been a combination in a unit test, a contract between services, realistic data, or a response to a failure that was already visible. Add a new check at the lowest level that reliably detects the cause, and add a higher-level check only when it verifies an important connection.

Editing and removing tests is equally important. A scenario without a current risk, clear expectation, or owner does not create confidence merely because it is green. Document the decision to retire it and check whether another test or a deliberate manual measure covers the same risk.

When the evidence is sufficient for a release decision

A green icon is not enough on its own. Before making the decision, know which version and environment were tested, which risks were covered, which tests did not run, and whether retries concealed the first failure. Open defects should have a known impact and a named decision owner.

A red result does not automatically mean a product defect either. The cause may be the test, data, or environment, but someone must diagnose it. A release can proceed according to agreed criteria and residual risk, not simply because the team lacks time to investigate the result. The guide to automated tests in CI/CD explains how to distribute checks across pull requests, deployments, and releases.

When the suite takes too long or the team does not trust it

For a long suite, first separate queue time, environment preparation, test execution, fixed waits, and retries. Then move combinations to a lower level, isolate data, parallelise safely, remove duplication, and divide feedback by risk. Simply disabling slow tests without a replacement only hides uncovered risk. A detailed process is available in the guide to speeding up a regression suite.

If the team does not trust the results, start with a small critical layer. Track the first-attempt result rather than only the final status after a retry, and distinguish the application, test, data, and environment when anything fails. Quarantine should be temporary, with a reason, owner, and remediation date. A growing number of skipped tests or green outcomes after retries signals that the suite is losing decision value; an audit of why test suites stop helping can guide the recovery.

What you gain

Layered regression testing provides a fast signal for a small change and wider evidence where the risk is greater. The team knows why a particular test ran, what its failure means, and who should respond. Such a suite supports a release decision without pretending that any number of green tests proves the absence of every defect.

Next step

Choose the most recent release or incident and record on one page what changed, which other areas could have been affected, which tests ran, and which decision their results influenced. Then divide the existing tests into fast, wider, and full layers. For each layer, add its trigger, maximum acceptable feedback time, failure owner, and the risk that remains outside it.