Performance tests

What to measure in a load test besides response time

Response time matters, but it cannot show by itself whether a system works correctly under load. An application may respond quickly because it rejects requests, returns incomplete data or moves work into a growing background queue. A report containing one average can therefore look good at the exact moment users can no longer complete a purchase.

The purpose of a load test is to describe the relationship between the amount of work, the user outcome and system behaviour. This requires several layers of metrics on a shared timeline.

Define the load precisely first

Metrics have meaning only in the context of a particular workload model. A p95 value at 20 requests per second does not answer what the system can handle at 200. Record the following before the test:

• concurrent virtual users or the arrival rate of new iterations;
• the mix of journeys, such as browsing, search, login and ordering;
• data volume and diversity, cache state and application version;
• the duration of warm-up, steady measurement and ramp-down;
• the environment, infrastructure limits and deliberate differences from production.

Without these details, two measurements cannot be compared fairly. It is equally important to verify that the load generator was not itself saturated. If it runs out of CPU, network capacity or connections, the measured ceiling may belong to the generator rather than the application.

1. Successful business operations

The first question is not “how quickly did the server respond?” but “how many users completed the intended action?” An HTTP 200 status does not necessarily mean success. An API can return an error state in its body, search can return no result for an existing product, or checkout can display confirmation without creating an order.

For each key journey, measure:

• the proportion of successful and unsuccessful iterations;
• errors grouped by type, endpoint and journey stage;
• response content and creation of the expected record;
• completed business transactions, such as orders per minute;
• operations that succeeded only after a retry.

In k6, checks record whether a particular condition is true, but a failed check does not necessarily stop or fail the entire test. The k6 documentation distinguishes checks from thresholds: a check records an outcome, whereas a threshold determines whether the test criterion passed. The principle applies to other tools as well—a business result needs both measurement and an explicit acceptance boundary.

2. Response-time percentiles instead of one average

An average hides the slow tail. Monitor at least the median, p90 or p95 and, for critical journeys, p99. The p95 percentile is the boundary not exceeded by 95% of recorded requests in that measurement. It does not automatically represent the share of satisfied users, because one person may make many requests.

Split response time by endpoint or business step. An overall p95 mixes fast static requests with a slow payment and can hide a problem in the most important operation. Where the tool provides timing phases, distinguish DNS, connection, TLS, time to first byte and body download.

Evaluate percentiles together with errors. Beyond capacity, response time can paradoxically improve because the application begins returning 429 or 503 immediately. The article on p95 versus average response time explains this interpretation in more detail.

3. Throughput and completed work

Throughput describes the requests, iterations or business transactions completed per unit of time. Be explicit about what is counted. One thousand HTTP requests are not one thousand orders, because one order may call several services.

Compare the requested and achieved rate. If a test schedules 100 new iterations per second but the generator starts only 70, it did not create the planned user load. If the generator sends requests correctly but completed operations stop increasing, the system has probably reached a capacity limit or is accumulating work in a queue.

The built-in k6 metrics separate iterations, requests, failures and duration. Whatever the tool, a report should connect technical throughput to business output.

4. Concurrency, waiting and queues

The same requests-per-second rate can have a different impact depending on how long work remains in progress. Monitor active requests, open connections and in-flight tasks. Increasing concurrency with unchanged throughput suggests that work is waiting.

Queues deserve dedicated metrics:

• queue length and the age of the oldest message;
• time waiting before processing and processing duration;
• retries and messages in a dead-letter queue;
• the difference between intake and completion rates.

A short test can look successful while unfinished work accumulates every minute. Over a longer run, the queue may consume memory or delay a customer email by hours. An asynchronous system cannot be judged only by the speed of the API that accepted the request.

5. Resource utilisation and saturation

CPU at 80% is not inherently a failure. What matters is whether a resource reaches its limit, creates waiting and degrades the user outcome. Monitor both utilisation and saturation:

• CPU, load, container throttling and CPU wait;
• memory, heap growth, garbage collection and out-of-memory restarts;
• disk operations, storage latency and IOPS exhaustion;
• network throughput, retransmissions, packet loss and connection limits;
• thread pools, workers and connection-pool occupancy;
• instance count and autoscaling response.

A metric without its limit is hard to interpret. Fifty database connections may be safe with a limit of 200 and critical with a limit of 50. Add capacity and waiting demand to a dashboard instead of displaying only the current value.

6. Database, cache and external dependencies

The final response combines work from several components. For a database, monitor query duration, locks, active connections, pool waiting, large scans and replication. For a cache, track the hit ratio, misses, evictions and the time required to load missing data.

For every important external service, record call count, error rate and response-time percentiles. One user request can trigger three retries after a failure and multiply load on an already slow dependency. Without separate metrics, the report shows only a slow checkout—not the payment call, connection pool or retry storm behind it.

Correlation in time does not prove a cause. CPU and p99 rising together creates a hypothesis. Confirm it with logs, tracing, profiling or a repeated test after a targeted change. The common web and API bottlenecks provide a useful investigation list.

7. Stability over time and recovery

A short test reveals immediate capacity, not slow leaks. During a soak test, watch memory trends, database and queue growth, open connections, scheduled jobs and gradually worsening percentiles. A stable average can hide a saw-tooth pattern caused by garbage collection or periodic pool exhaustion.

Measure recovery after load ends. How long until queues drain, instances return to normal and the error rate falls? A system that survives the test but cannot serve ordinary traffic for another 40 minutes needs a different assessment from one with the same peak and rapid recovery.

Thresholds must express a decision

A dashboard shows what happened. Thresholds determine whether the result is acceptable. One might say: “at target load, at least 99.5% of checkouts complete, p95 for the payment step stays below the agreed value and the age of the oldest message remains below its limit.”

The k6 threshold documentation describes expectations as pass/fail criteria. A threshold should not be an arbitrary number copied from elsewhere. It should follow user needs, business risk, an SLA or comparable production experience.

Distinguish target load, a boundary test and an intentional stress test. Degradation is expected during stress testing; success may mean controlled rejection without data corruption and subsequent recovery, not the same timing as normal operation.

What a useful report looks like

The report should combine the workload model, system version, thresholds and time-series charts. On one timeline, show arrival rate, successful transactions, errors and percentiles. Add resources, queues, the database and dependencies in adjacent panels. Mark test-phase changes, scaling events, deployments and incidents.

The conclusion should say more than “p95 was 1.8 seconds”. State the load at which a queue began to grow, which business operation failed first, which limit was exhausted and what the next test should verify. If the cause is not confirmed, label it as a hypothesis.

What the team gains

Layered measurement turns a load test from a response-time chart into decision support. The product owner sees which user journeys complete at target load, developers receive a concrete bottleneck hypothesis, and operations knows which limit was exhausted and how long recovery took.

The same metric set also compares two system versions without relying on impressions. The team can determine whether an optimisation made a critical operation faster without raising errors, shifting work into a queue or consuming disproportionate infrastructure.

Next step

Add three layers to one existing scenario: business-operation success, throughput and at least one saturation metric for the likely bottleneck. Set thresholds before execution and verify that the generator achieved the planned load. This turns a timing measurement into a test that supports decisions about capacity, release risk and further diagnosis.