Tests pass locally but fail in CI: how to find the difference
A test that passes on a laptop and fails in CI is not contradicting itself. It ran under two different sets of conditions. The purpose of diagnosis is therefore not to repeat it until it turns green, but to discover which difference changes the result.
Why local and CI results differ
Continuous integration (CI) builds software and runs checks on a separate runner, the machine or process that executes automated steps. That runner may have a different operating system, capacity, time zone, browser version, access rights, and test data from the local computer. The more accurately you capture those conditions, the easier it becomes to reproduce and fix the failure.
Preserve evidence from the original failure first
Do not begin with another run alone. Retain the test name, commit and application build, exact command, execution time, runner, browser, and first error message. Add the test’s console output, relevant application and network logs, and a request or correlation ID if the system uses one. The first cause matters; later errors may only be consequences.
For UI tests, a trace, screenshot, or video may help. They are not automatically available for every tool or run. Playwright, for example, keeps recording off until its configuration enables it, with options to retain output only on failure or retry. Its Trace Viewer can then show individual steps, page state, console output, and network activity.
The artefact must belong to the failed attempt. A video from a repeated attempt (retry) that passed may not reveal the original problem. Recordings may also contain personal data, tokens, or completed forms, so give them restricted access and a suitable retention period.
A realistic example: an order fails only in the nightly CI run
Consider a test that creates an order using today’s date for delivery. A developer runs it at ten in the morning in Bratislava and it passes. The CI runner uses UTC, the test starts shortly after local midnight, and the API now calculates a different “today” from the browser. A screenshot shows only a wrong date in the confirmation; without time-zone information, the failure looks like an intermittent UI problem.
Four workers, or concurrent execution processes, run in the same suite. Each uses e2e-customer@example.test and empties that account’s basket before its test. One scenario can therefore remove another scenario’s items. Only one worker runs locally by default, so the collision does not occur. The local machine also has an older browser in its cache, while CI performs a clean installation and uses the version determined by its dependency lockfile and runner image.
This is not one “CI bug” but three separate hypotheses: time, concurrency, and environment version. A sound investigation does not change all three together. It first repeats the CI run with one worker, then aligns the time zone, and finally compares exact versions. This identifies which condition controls the outcome and which differences were incidental.
Build a systematic environment diff
Instead of changing timeouts at random, compare the local and CI runs across the same categories:
- Code and command: the same commit, built artefact, working directory, test tags, and configuration files. Confirm that the local command really is the one used by the pipeline.
- Versions: runtime, packages resolved by the lockfile, browser, drivers, operating system, and container image. For a remote Playwright server, the official documentation says the version in the tests should match the version running in the container.
- Configuration: environment variables, feature flags, base URL, proxy, certificates, headless mode, and permissions. Log names and safe fingerprints of settings for diagnosis, not password values.
- Time and locale: time zone, locale, date format, system clock, and scenarios near the start of a day or month. A browser may also have its own emulated locale or time zone.
- Data and access: database migrations, seeded records, account state, role, token expiry, secret availability, and external-service behaviour. A missing secret may surface as an ordinary sign-in failure.
- Resources and network: CPU and memory capacity, latency, DNS, ports, firewall rules, and application start-up time. A slower runner often exposes a wait tied to elapsed time instead of the required state.
Write the result as a short environment diff: what matches, what differs, and what remains unknown. Assumptions then become hypotheses that the team can test.
Reproduce the conditions, not just the test
Run the failing test from a freshly retrieved repository (clean checkout) with the same lockfile and command. If the pipeline uses a container, try the same image locally. Conversely, a diagnostic CI step can print safe tool versions, locale, and time zone. The official Playwright CI guide, for example, separates package installation, browser installation, and test execution, allowing each phase to be checked.
If the test passes alone but fails in the suite, investigate concurrency and shared state. Temporarily set one worker and compare the outcome. Then look for the same account, record name, file, port, or assumed test order. One worker is a diagnostic experiment rather than the final repair; the goal is to isolate data so tests can run concurrently without collisions.
If it also fails alone, change one condition at a time: time zone, version, configuration, or data source. Record the outcome of each attempt. When failures are unpredictable in both environments, you are more likely dealing with instability that should be measured across run history, not only a local-versus-CI difference.
How to confirm that you found the cause
One successful run is not evidence on its own. A hypothesis becomes convincing when you can control the outcome: under the original condition the test repeatedly fails, and after one targeted change it passes without other interventions. For an account collision, keep the application and test code unchanged but give every worker a separate account. Then repeat the suite with the worker count used in CI.
Design a small experiment for each hypothesis:
- write the expected outcome, such as “the collision disappears with one worker”;
- identify the single variable to change and freeze everything else;
- repeat both the failing and corrected configurations several times;
- check that the failure did not merely move or change type;
- add an automated check that will reveal a recurrence.
The final point separates diagnosis from a durable repair. If a missing secret caused the problem, the pipeline should fail during validation of required configuration, not five minutes later in a sign-in test. If the time zone was responsible, set it explicitly or exercise boundary dates with a controlled clock. If shared data caused the collision, generate identifiers from the run and worker rather than leaving the suite serial.
After the repair, compare at least these pieces of evidence: the same test passes from a clean local checkout and in CI, passes alone and in the complete suite, and produces diagnostic output containing the decisive versions and configuration. For a concurrency problem, execute the target group with the intended parallelism instead of only once in serial. For a time-related problem, exercise the boundary where it originally failed.
Record the cause in the incident or work item as “condition → mechanism → symptom”. For example: “Two workers changed the same basket, so one removed an item before the other’s assertion and the test reported the wrong total.” This is more useful than “increased timeout” and can become a rule for the entire suite.
Repairs that hide rather than solve the problem
A longer timeout can test a hypothesis about a slow operation, but it should not remain the only repair without checking the actual state. A retry may collect more evidence, but a pass on the second attempt must not erase the first failure. Likewise, do not leave the pipeline permanently serial if shared data is still the real problem.
Avoid changing application code, the test, the runner, and data at the same time. The result may turn green, but it will not show which change removed the failure or whether it can return.
What you gain
A reproducible difference shortens the path from a red result to its cause. The team can separate an application defect from a test defect, missing configuration, or a data collision. It also gains a checklist of information every CI run should retain, so the next failure does not begin without context.
Next step
For the next failure, create a one-page comparison of the command, versions, configuration, time, data, and worker count for both runs. Then transfer only one difference into the local environment or CI at a time. If the suite has been unclear for a long time and lacks useful evidence, the process for taking over an inherited failing suite can first restore control over execution and ownership.