Fully autonomous testing: what AI can and cannot do
Fully autonomous testing describes a system that would select scenarios, create tests, run them, evaluate the results and update them after an application change by itself. AI can already support individual parts of this process. Combining them without human oversight is risky, however, mainly because the system needs a trustworthy definition of the correct result and the business priority.
Why it is appealing
Writing and maintaining tests takes time, and neglected suites can become outdated and stop helping the team. Automatic generation, result classification and suggested repairs therefore have practical value. The problem arises when accelerating mechanical work is confused with taking responsibility for quality.
Where it reaches its limits
The most important limitations are not only related to model quality. They also arise from missing context, risk measurement and accountability for decisions.
1. How does the system know the correct result? A test needs an “oracle”: a trustworthy definition of the expected result. For an order, that may include the correct status, amount, items and dispatch of a confirmation. If these rules are not present in requirements, data or another verifiable source, the model can only guess them. Calculating a 25% discount instead of 20% may not cause a technical error even though it is wrong for the business.
2. What should be tested at all? Judging value and risk Deciding which scenario is critical requires information about revenue, the impact of a defect, frequency of use and contractual obligations. Without it, a model may assign similar importance to signing up for a newsletter and completing a payment. We also describe this boundary in the article on whether AI will replace software testers.
3. Unreliability: tests that pass when they should not Autonomous maintenance may suggest redirecting a test to a similar element even though the change was caused by an application defect. A model can also create a test without sufficient assertions. A green result alone therefore does not confirm that the suite protects important behaviour.
4. A loop in which AI repairs AI: drift If a model repeatedly modifies tests without comparing them with a requirement or approved behaviour, drift may occur: the tests gradually move away from their original purpose. Changes therefore need an external anchor, such as acceptance criteria, approved business rules or human review.
5. The illusion of coverage A large number of generated tests does not automatically mean good coverage. What matters more is whether the suite covers critical user journeys, rules and edge cases, and whether it can detect a relevant defect.
6. Instability at scale Unsuitable selectors, waits and test data can cause intermittent instability. A high volume of generated tests also increases the number of results someone must evaluate.
7. Costs that are difficult to predict If the model is used for every draft, evaluation and repair, its usage and the total cost of the solution increase. The estimate must also include the human effort required to resolve ambiguous cases.
8. Security and data With a cloud-based model, application content may leave your infrastructure. The broader the autonomous system’s access, the more important an isolated environment, minimum permissions, permitted domains and an audit trail become.
9. Accountability The team needs to know who approves scenarios, automatic repairs and the decision to release. In regulated or financially sensitive areas, a traceable human decision is often as important as the test itself.
How to use autonomy safely
A practical collaboration model divides responsibility as follows:
- a person defines the correct result, priorities and access boundaries,
- AI prepares a draft, repeats approved patterns and classifies results,
- an automatic repair is created as a proposal with a record of the change,
- a tester or risk owner approves the proposal before it is added to the main branch,
- the team measures detected defects, stability and maintenance time, not only the number of tests.
During creation, AI may have controlled access to the test application. Once approved, the output remains standard Playwright code in your repository that can run without the model.
What you gain
- Faster preparation and maintenance of repetitive parts with clear approval controls.
- Better traceability of why a test or its repair was created.
- A suite in standard code that can be maintained without a particular model.
Next step
Before a pilot, define which decisions the system may only propose and which it may perform automatically. If you would like to set the boundaries according to your product’s risks, use a no-obligation consultation.