Test strategy and audit

Smoke, sanity, and acceptance tests: what they mean and when to run them

Smoke, sanity, and acceptance tests answer different questions, although teams often confuse them. Smoke asks whether a build or deployment is testable, sanity is often a narrow check of a change, and acceptance assesses alignment with criteria and a user or organisational need. Each suite still needs a defined objective, scope, inputs, owner, and decision.

Why the same terms are used differently

These labels are not three mutually exclusive technical folders. They primarily describe a run’s purpose. The same automated purchase scenario can form part of a post-deployment smoke suite, a pre-release regression suite, and a user story’s acceptance check. Its trigger, required evidence, and response change.

“Sanity” causes the most confusion. Some teams use it for a narrow check of a corrected area, some for a small regression subset, and others almost as a synonym for smoke testing. It is therefore unsafe to copy the name from another project and assume it has the same meaning. This article uses a common practical distinction: smoke is a broad and shallow check that a build or deployment can be tested, sanity is a narrow check of the changed area, and acceptance determines whether the solution meets the agreed need.

Smoke testing: can this build be tested meaningfully?

A smoke test performs a small number of basic checks across the system. It should quickly reveal that a build will not start, a service does not respond, a database migration has failed, configuration points to the wrong dependency, or a critical journey cannot be opened. It is broad because it touches important parts and shallow because it does not exercise many combinations or detailed boundary cases.

After an online shop is deployed, its smoke suite might check that the catalogue loads, sign-in works, a product can be added to the basket, checkout opens, and the order API accepts a controlled request. It should not prove the correctness of every discount, delivery method, and payment option. A green smoke result means “the environment and basic journeys are ready for further testing”, not “the release is safe”.

The ISTQB Advanced Level Test Analyst syllabus also uses a smoke test to verify the readiness of a test environment. In practice, teams can run one against a local build, an integration environment, or a production deployment. Scope changes with the environment: in production, scenarios should be controlled and non-destructive or safely clean up their data.

Sanity testing: does the changed area work well enough to continue?

A sanity test is often used after a small change, defect fix, or new build for a narrow check of the affected feature and its immediate surroundings. If collection-point selection has changed, the sanity suite might check loading and choosing a point, preserving that choice while moving through checkout, and storing its identifier in the order. It does not test the entire shop.

Its boundaries partly overlap with confirmation testing, or retesting a fix, and with targeted regression. The team needs to state the distinction. It might agree that confirmation testing always reproduces the original defect, while the sanity pack checks the basic stability of the whole changed area. Another team may avoid “sanity” altogether and use “targeted regression”. Either approach works if everyone understands the scope from the result.

There is no universal size or time limit for a sanity suite. It should answer early yet be deep enough to support proceeding with wider testing. If it becomes an hour-long check of unrelated features, it no longer fulfils that purpose even if it retains the name.

Acceptance testing: does the solution meet the agreed need?

Acceptance testing validates whether a solution meets the conditions under which the relevant stakeholders will accept it. The test basis may be a user story’s acceptance criteria, business rules, operational requirements, a contract, or regulatory conditions. A check may be manual or automated, and it can start during development if the team turns the criteria into executable examples.

Acceptance is therefore not limited to user acceptance testing (UAT), in which intended users exercise a finished system. The ISTQB Foundation Level syllabus lists user, operational, contractual, and regulatory acceptance testing, as well as alpha and beta testing, among its forms. Only some of these may be relevant to a particular product.

An example of a functional acceptance criterion is: “When an order contains only goods eligible for a collection point and the address is in a supported country, the customer can choose a collection point and its name appears in the confirmation.” Operational acceptance could additionally check that the collection-point code reaches the warehouse, support can see it in administration, and a failed integration can be traced. A successful feature demonstration does not replace this evidence.

A side-by-side comparison

The table is a starting point, not a universal standard for pipeline job names. If a team uses “sanity” differently, it should adjust its own glossary and add a more precise description to the name, such as checkout-change-sanity.

The inputs and outputs each run needs

Without clear inputs, scope changes depending on who started the test. Without a clear output, a report is produced but nobody knows what to do with it.

Smoke needs a build and environment identifier, available basic dependencies, controlled test data, and a list of critical probes. Its output should state which components were available, where a scenario stopped, and whether further testing can proceed. When it runs after deployment, the team needs to distinguish an artefact defect from an environment configuration problem.

Sanity needs the change description, affected components, original defect or acceptance examples, and expected adjacent dependencies. Its output should be more than “5 of 5 passed”; it should also confirm the tested version and boundaries. For example, collection-point delivery works, but courier delivery was not part of this run.

Acceptance needs approved and testable criteria, the correct roles, representative data, and someone authorised to evaluate the result. Its output is evidence against the criteria, deviations, open risks, and a decision. A screenshot may be a useful artefact, but it does not demonstrate processing in the API, database, or a downstream system.

Where they belong in the pipeline and release process

A typical order looks like this:

1. The build passes static, unit, and component checks.
2. After deployment to a test environment, a smoke test confirms that the environment and critical parts can be tested.
3. Sanity testing or targeted confirmation testing checks the change and its immediate surroundings.
4. A wider regression suite looks for unwanted effects in other areas.
5. Acceptance tests confirm the specific criteria and readiness from the relevant stakeholders’ perspectives.
6. After production deployment, a safe smoke test can confirm the deployed configuration and availability of critical journeys.

This order is not a rigid phase model for every project. Automated acceptance examples may run on a pull request, while operational acceptance needs an assembled system in a representative environment. A team can omit sanity testing if targeted regression unambiguously answers the same question. The guide to what to run in CI/CD provides a practical distribution of test runs.

One online-shop release, three different kinds of evidence

An online shop is adding delivery to a collection point. After deploying the candidate, the smoke suite loads the catalogue, adds a product to the basket, opens checkout, creates a controlled order, and checks the availability of the order API. If checkout fails because carrier configuration is missing, further testing stops and the team investigates the deployment.

After the green smoke run, the sanity suite targets the new feature: the list of points loads for supported postcodes, the choice is preserved when moving back through the steps, the fee is recalculated, and the point identifier is stored in the order. If the last check fails, the build is generally testable, but the change is not ready for wider verification.

Acceptance tests then cover every agreed criterion, including ineligible goods, the customer confirmation, visibility in administration, and the warehouse export. The product owner evaluates functional criteria, operations evaluates the data flow, and the tester prepares evidence and boundary cases. Separate regression testing still checks that existing courier delivery, discounts, and payment continue to work. None of the three green checks replaces the others.

Who should own the tests

The owner does not have to execute every scenario personally. They are, however, accountable for scope, trustworthiness, and response. Developers, testers, and the platform team often maintain smoke testing together because a failure may be in either the application or deployment. Sanity should sit as close as possible to people who understand the change. For acceptance, it must be clear who has the authority to confirm business, user, operational, or other criteria; a tester can guide the process but should not invent the product need alone.

Record a deputy owner and escalation path for each suite as well. An automated job without a person to evaluate a red result is not a release control. Equally, it is not enough for a product owner to “sign off UAT” formally without knowing the uncovered criteria and residual risk.

What to watch out for

• A smoke suite that tries every combination will deliver its basic signal too late. Move detailed rules to an appropriate layer according to the test pyramid.
• Sanity without a team definition creates false expectations. Record its question, boundaries, and difference from confirmation and regression testing in the README or strategy.
• Acceptance without measurable criteria becomes a subjective demonstration. Criteria should include negative and operational scenarios when they matter to acceptance.
• A green post-deployment smoke test does not replace monitoring. It confirms specific controlled journeys at a particular time, not the continuous experience of every user.
• A retry must not silently turn an unclear first failure into a trustworthy green result. Retain the first-run outcome and determine the cause of the instability.

What you gain

Precise labels shorten the release discussion: the team knows whether it is dealing with an untestable build, a problem in the changed area, an unmet criterion, or a regression outside it. Each run can have proportionate scope, and its result reaches someone with authority to act. Instead of one vague “the tests passed”, the team obtains separate pieces of evidence with explicit boundaries.

Next step

Open the configuration of the most recent pipeline and add five items to every job: the question it answers, its trigger, inputs, owner, and the decision after a green or red result. Then mark any jobs named smoke, sanity, or acceptance whose contents do not match that definition. Rename them or adjust their scope before using them to decide on the next release.