Automated tests in CI/CD: what to run on pull requests, after deployment, and overnight
CI/CD automates how software is built, checked, and deployed. Adding tests does not mean running the entire regression suite for every change. A useful design gives fast feedback on a pull request, checks more before release, and produces a clear signal after deployment without letting an unstable suite block the team.
Why one test job is not enough
Tests differ in speed and the evidence they provide. A unit test checks a small piece of code, an API test verifies logic through an interface, and an end-to-end (E2E) test exercises a complete user journey. Accessibility and performance checks have different conditions again.
One long job delays feedback and complicates diagnosis, while only a few fast checks may miss a serious defect. Divide the suite by risk and by when its result can still influence a decision.
A practical execution matrix
The following matrix is a starting point, not an immutable rule. A critical financial operation may need stricter checks than a content page, while a small monolith may use a different pipeline from a set of microservices.
On a pull request: optimise for fast feedback. Run unit and component tests, static checks, relevant API tests, and a small UI smoke pack. A smoke test briefly checks key journeys such as signing in. Add automated accessibility checks for changed components. The objective is to stop an evident regression before merge, not prove the whole system correct.
After merging to the main branch: broaden integration coverage. Add more API and integration tests, critical E2E scenarios, and accessibility checks for important states. Verify the built deployment candidate. If this gates the next stage, the tests must be stable and failure handling must be clear.
After deployment: test what was actually deployed. Confirm that the application responds, a user can sign in, and one or two critical journeys work. In production, use controlled accounts and non-destructive scenarios or reliable cleanup. The purpose is to catch configuration, routing, secret, or dependency problems—not repeat the complete regression suite.
Overnight: gain breadth without making a developer wait. Run broader E2E journeys, more browsers, a larger permissions matrix, accessibility checks, and scenarios that use slower external systems. Give the result an owner and a morning triage process; ignored failures are not quality control.
Before a release or campaign: test the specific risk. Run the risk-ranked regression suite and change-related scenarios. Performance tests need a separate, controlled run with an agreed environment, load, and thresholds—not a shared runner with routine UI tests. The comparison of load, stress, spike, and soak testing helps select the workload.
Where each test layer belongs
- Unit and component tests should form the fastest layer and run for every relevant change.
- API and integration tests cover business rules without browser overhead. Run a fast subset on pull requests and more after merge; our REST API testing guide describes practical checks.
- UI/E2E tests are most valuable for key journeys. Remove fixed waits and shared-data dependencies before adding them to the pipeline.
- Automated accessibility checks should cover specific states such as an open dialog. A scanner does not replace human assessment; account for its limitations.
- Performance tests should be separated by risk: a small baseline test may monitor a regression, while a capacity or stress test requires dedicated conditions and coordination.
Which failure should stop the pipeline?
A quality gate is a rule that must pass before the pipeline continues. Base it on a trustworthy signal with a clear owner, such as a unit failure, broken API contract, or failed critical journey. An unproven check may initially warn rather than block.
Do not confuse a retry with a fix. One retry can collect diagnostics and distinguish a repeatable failure from instability. If a test passes only on retry, label and track it instead of reporting a clean success. First distinguish application, test, and environment problems. Give any temporary quarantine an owner and deadline.
Diagnostics, parallelisation, and security
On failure, retain the report, logs, application version, and environment details. UI screenshots, videos, and traces may contain personal data or tokens, so restrict access and retention. The official Playwright CI guide shows runs on pushes, pull requests, and successful deployments, plus HTML report storage.
First isolate tests that could modify each other’s data. Then use parallel workers or sharding, which divides a suite across CI jobs. Playwright supports sharding and merged reports, but extra workers can overload the environment or create shared-account conflicts.
Keep passwords, API keys, and tokens out of the repository. Use the CI secret store, least-privilege accounts, and masked output, such as GitHub Actions secrets or protected GitLab CI/CD variables.
What you gain
Separating tests by risk and execution point gives the team a fast signal for a small change, broader confidence before a release, and a distinct confirmation after deployment. It also makes it clearer which layer failed and who should respond. The team does not need to wait for the full regression suite on every pull request or learn to ignore one long job that is frequently red.
Next step
List each test’s duration, stability, and covered risk. Create a short pull-request pack and post-deployment smoke test, then define gates and failure ownership. Add broader runs when every red result has a clear path to diagnosis and a decision.