Testing web applications

How to accept automated tests from a supplier and know they work

A supplier demonstrates a successful automated test run and hands over a code repository. An engineering lead, product owner or test lead also needs to know what the tests check, whether their team can run them and how they will recognise a problem. We therefore recommend combining a review of the agreed scope with a practical exercise in using the tests. The outcome should support a decision on whether the delivery meets the agreed conditions and what still needs work.

Why a successful demonstration is not enough

A successful result can look convincing even when a test has only opened a page, filled in a form and pressed a button. If it never compares the expected outcome with the actual one, that demonstration provides no evidence that the completed order is correct. Nor does it show that you can run the same suite with your own access credentials.

We suggest preparing acceptance as a working session using a specific version of the delivery. Agree in advance on the scenarios, environment and evidence you will review. The supplier then knows what to prepare, and the client assesses the same conditions that were agreed during implementation.

Define coverage that you can verify

A list such as “login, basket, order” is a useful conversation starter but a weak basis for acceptance. For each area, record the user situation, starting conditions and expected outcome. For login, for example, distinguish valid credentials, an incorrect password and an account without the required permission, where these are within scope.

Assign a test or test group identifier to each agreed situation. During the review, you can then follow the requirement to an executable scenario and on to its result. The number of test files is supplementary information: one file may contain several checks and several variations of the same case.

If the scope changes during the work, update this list too. Mark any skipped scenario and explain its impact. Acceptance may include an agreed exception, but the person reading the result needs to know which risk remains unchecked.

Check what the test actually compares

A check that compares reality with an expectation and flags a mismatch is called an assertion in test code. In a web test, it may verify confirmation text, an item count or the status of a stored record. Ask the supplier to use a selected scenario to show the connection between the requirement, the specific check and the message produced when it fails.

For example, Playwright provides text, visibility and value checks through expect. It repeats selected checks on web elements until the condition is met or the time limit expires. Distinguish this waiting for an outcome from rerunning the entire test. Playwright assertion documentation.

A visible confirmation may not satisfy the complete requirement. If the scope includes storing an order with the correct items, we suggest checking those details through the agreed interface as well. The technical implementation can vary; what matters is checking the property that the test was created to protect.

Have the client’s team run the tests too

For the exercise, use the client’s working environment and the agreed procedure the team will use after handover. If this involves running tests from the repository, a designated colleague obtains the agreed code version, prepares the dependencies using the instructions, configures access and runs the selected suite. The supplier can explain uncertainties; add missing steps to the documentation as you go.

If the tests are to run through a web interface or scheduler, have the team try that route, including the agreed environment and data settings and access to the results. Check the required permissions and availability of instructions for this method too.

Also verify a repeat run after resetting the data. A test that needs a manually prepared account or record may be usable if that condition is agreed and documented. An acceptance problem arises when there is an unknown dependency that only the author knows how to provide.

If the delivery includes execution within an automated build and deployment process, known as CI/CD, try that route too. A standalone laptop run does not replace this part of the scope. For every agreed method, check that the report is available and that the relevant system visibly signals an unsuccessful result.

Illustrative example: checking an order failure

The following situation is illustrative, not a result from a particular client. A team is accepting a web test for an order containing two units of a test product. The agreed checks verify the quantity and the accepted order status. Payments, dispatch and messages to real customers are outside this exercise and safely separated in the environment.

First, let the scenario pass with correct data. Then agree with a developer on a temporary response change for this scenario alone in an isolated test environment: the response will contain one unit instead of two. Keep the expectation in the test unchanged. Verify that the test fails on the quantity check and that the report shows the difference between the expected and actual values.

This exercise checks how a specific assertion responds to an incorrect result. Because it uses a modified response, it does not by itself confirm that all real services work together correctly. Verify that separately through an agreed scenario. If a suitable change cannot be safely isolated, choose another check or a separate test instance with the supplier.

After the exercise, remove the temporary change, reset the data and repeat the original scenario. Save both results and a description of the change in the acceptance record. Make changes only in the agreed test environment using non-production data.

A failure must give the team useful evidence

For an unsuccessful run, check whether a colleague can find the scenario name, failed assertion, expected and actual results, run time and application version without the author’s help. A detailed record of the test execution, or log, should help determine the next investigation step. The report need not decide by itself whether the fault is in the product, test or environment.

For a Playwright solution, a recorded execution known as a trace can supplement diagnostics. Trace Viewer lets you inspect recorded actions, errors and network requests; recording needs to be configured. Agree on the specific outputs for the solution being delivered. Trace Viewer documentation.

Check the results after retries too. Playwright distinguishes tests that pass first time, tests that pass only on a retry as flaky, and tests that still fail after retries. Test retry documentation. During acceptance, make the original failures visible and agree how to address instability even if the last attempt passed.

The exercise should also include opening the outputs as an ordinary authorised team member. Agree on their location, retention period and access. Use test data and check that the outputs contain no authentication secrets.

Documentation that supports practical work

Hand over instructions with the code and test them during independent execution. They should cover the required tool versions, installation, configuration, data preparation, running the entire suite and a selected scenario, and opening the results for the agreed method of use. Password values do not belong in the instructions; describe how to obtain them securely.

A technical colleague should also be able to find where to add a check or adjust an expectation following an agreed application change. A short description of the code structure and dependencies helps. Judge the documentation by whether it enables someone to complete a routine task, rather than by its page count.

Acceptance and subsequent maintenance

During acceptance, record the specific code version, environment, reviewed scenarios, evidence and outstanding issues. For each issue, identify its impact, a responsible person and the next decision date. If an agreed essential check is missing or the client cannot run the suite as agreed, record this as an unfinished part of the delivery.

Future changes to the application, data and dependencies need a separate maintenance agreement. Distinguish completing unmet requirements from new requests after acceptance, and agree how to track them. The article on who owns automated tests discusses the ongoing division of responsibilities in more detail.

Checklist and first step

Prepare this short checklist for the session:

• Every agreed situation has a traceable test and an expected outcome.
• The selected checks reflect the actual purpose of the scenario.
• The client’s team has run the agreed suite using the supplied instructions.
• The requirements for preparing and resetting data are known.
• A safely induced failure appeared in the relevant check and report.
• Retries and skipped tests are visible and explained.
• An authorised team member can open the outputs and perform an initial assessment.
• Outstanding issues and the boundaries of subsequent maintenance are recorded.

This acceptance process gives the team a verifiable basis for using the delivery: they know the scope of the checks, can run the tests and have evidence to investigate failures. As a first step, select one important scenario and agree with the supplier how to demonstrate both a successful run and a deliberately unsuccessful one. Use this example to check whether your acceptance conditions are specific enough.