Performance tests

Load testing in staging or production: what can be measured safely

A load test in staging reduces direct risk to customers, but staging often lacks production capacity, data, or dependencies. A test in production uses the real system, yet it can affect users and create genuine business consequences. The right choice is therefore not a contest between two environments: each answers a different question and needs different guardrails.

The problem: the environment is part of the result, not just an address

“The system handled 500 users” has little meaning without conditions. The application version, infrastructure size, cache state, data volume, autoscaling configuration, external-service behaviour, and load-generator location all affect the result. The same script can also produce a different result when run at another time.

The simple rule is that a result applies under the tested conditions. You cannot automatically calculate production capacity from a measured staging capacity based on CPU count. Nor does a production test establish behaviour during every future peak when it covered only one scenario, duration, and load profile.

Example: an online shop before an advertising campaign

An online shop expects a sudden increase in traffic after a campaign is sent. The existing and new builds are deployed to staging in turn; the environment has one application instance instead of four, a smaller database, and a payment-gateway test environment, or sandbox. The team repeats the same scenario against both builds. Under the same load, the new version has a worse p95—the threshold not exceeded by 95% of measured responses—and database metrics show more slow queries. This is useful evidence of a performance regression under comparable conditions, not evidence of production capacity.

After a fix, the team repeats the comparison. The difference disappears, but production autoscaling remains an open question because staging does not have it. The system owner therefore approves a short production measurement outside peak hours: catalogue reads and dedicated test accounts only, gradual ramp-up, a fixed maximum request count, and stop conditions. The payment path remains excluded until its side effects and provider limits have been agreed.

The outcome is not one combined number. The staging report confirms that the regression has been removed; the production report describes the response of real scaling under a limited profile. The risk of a sudden surge across the complete purchase journey remains explicitly unverified until a safe experiment covers it in a representative environment.

What staging can measure well

Staging is useful for detecting a performance regression early, comparing two versions, and investigating a bottleneck without directly affecting live operations. The team can deliberately provoke a higher error rate, change configuration, and repeat the test after a fix. Grafana k6 explains that both pre-production and production testing add value, while warning that results from a test environment do not necessarily apply to production.

Before measurement, map the differences between the environments:

• the application version, runtime, and configuration that materially affect behaviour;
• service topology, instance counts, CPU and memory limits, and scaling policies;
• data volume and distribution, indexes, and warm or cold caches;
• databases, queues, and external dependencies, including their sandboxes and limits;
• the network path and location of the load generator;
• available application, database, and infrastructure metrics.

Not every difference needs to be removed, but it must be recorded and reflected in the question. Under sufficiently repeatable conditions, a smaller staging environment can show that a new build is 20% slower than the previous one under the same profile. It may not establish how many customers a larger production deployment can serve. The k6 guide to automated performance testing likewise recommends scaling the workload to staging when its scale differs from production.

When controlled production measurement makes sense

Production reveals the real configuration, data, network path, scaling, and dependency behaviour. It can cautiously confirm a critical path, test a scaling response, or answer a question that cannot be reproduced credibly elsewhere because the environments differ too much. A more representative environment carries greater risk, however; it does not automatically make every test better.

Such a run needs explicit authorisation from the system owner and agreement with operations. Record the following before it starts:

1. Goal and load profile. Which scenarios, request volumes, ramp-up pattern, and duration will be used.
2. Window and responsibilities. Who observes the test, who can stop it, and who responds to an incident.
3. Guardrails. The maximum load and response-time, error-rate, utilisation, or queue-length thresholds that trigger an automatic or manual stop.
4. Test accounts and data. How orders, emails, invoices, inventory changes, and analytics are separated from real customers and cleaned up.
5. External services. Which calls are replaced, limited, or agreed with a provider in advance so that the test does not exhaust quotas or cause an unintended effect.

Begin with a short smoke test, meaning a check at minimal load, and increase it only after reviewing the metrics. For production testing, Grafana recommends lower load levels, less risky test types, an appropriate time window, and well-prepared monitoring and diagnostics. A stress test that continues to the point of failure is not a routine production check; it belongs only in a separately approved experiment with an accepted impact and recovery plan.

Decide from the question and consequence of failure

To compare builds, configurations, or a database query, prefer repeatable staging and keep the other conditions unchanged. To determine absolute capacity, staging must be sufficiently similar to production in resources, data, and scaling; otherwise, label the result as staging capacity only. If the question concerns the real network path, production scaling, or live configuration, a small production test may be appropriate.

Do not ask only “where will the number be more realistic?” Also ask “what happens if the test exceeds expectations?” A scenario that can exhaust a shared database pool, send thousands of messages, or trigger invoicing needs isolation or a pre-production environment. Choose production only when the answer’s value outweighs the remaining risk and the team can stop the test before it causes an unacceptable impact.

What to measure in both environments

The number of virtual users alone is not enough. Monitor throughput, the error rate, and response times, including the p95 and p99 percentiles. Agreed limits can be expressed in k6 as thresholds, or pass/fail criteria for the test. This does not mean the chosen boundary is right for your product; it must come from business and technical expectations.

On the same timeline, monitor CPU, memory, database connections, queue lengths, caches, instance counts, and metrics from important dependencies. This creates a hypothesis about the likely bottleneck, which can then be tested with logs, profiling, or another run after a targeted change. Monitor the load generator too: if it reaches its own limit, it can create a false picture of the application’s capacity.

A combination is often more accurate than one large test

First refine the script, data, appropriate load-test type, and diagnostics in staging. Repeat a reference run to learn the normal variation. Then, if the risk justifies it, add a small production test that checks one specific difference, such as actual scaling or the network path.

The report should state the version, environment, scenarios and their proportions, load profile and duration, data and cache state, generator locations, thresholds, and events during the test. Keep these conditions as similar as possible when comparing two runs. Otherwise, an observed difference may come from the experiment rather than the application change.

Before drawing a conclusion, verify that the test actually produced the planned profile. Compare planned and achieved request counts, load-generator errors, and the shape of the load over time. Run the reference test more than once so that one fluctuation is not presented as a trend. If a stop condition ended the run early, that is an important safety result but not complete confirmation of the original goal. The report should state where measurement stopped and which questions it no longer answered.

What to watch out for

Live traffic creates noise that staging does not have: campaigns, batch jobs, backups, and a changing mix of user journeys. Staging may instead use simpler data or inactive integrations. Neither environment provides a “pure truth” without context.

Send load only to a system you own or have explicit permission to test. Even your own system may share a database, network, or external provider with other services. Confirm the scope of authorisation and guardrails before the test, not after the error rate begins to rise.

What you gain

Staging provides a repeatable place for diagnosis and comparison of changes. A controlled production test can add evidence about the real configuration. When both results include their conditions and limits, the team understands what was measured, what can be compared, and which risks remain open.

Next step

Write down the question the test should answer and a table of differences between staging and production. If staging is sufficient, measure the trend and bottleneck there. If only production can verify the decisive difference, prepare the smallest possible scenario, written authorisation, monitoring, stop conditions, and cleanup plan before starting the first virtual user.