Why the suite was rewritten and why AI was included from the start
The client had a working suite of automated web tests in Selenium. The objective was therefore not to introduce testing, but to rebuild the existing solution. The reason was an ongoing initiative to standardise testing tools across the company so that every team would not use something different. One tool means one set of rules, one body of expertise and one set of procedures worth refining properly.
Once a suite is being rewritten, it is also the best time to incorporate AI. The intention was to build it so that AI would take over the most time-consuming part of the work: writing and maintaining tests. Rewriting the entire suite manually would have taken months.
For this migration, Playwright provided a suitable modern browser-automation foundation for the target Robot Framework stack. For another project, the choice should reflect the existing suite, team and infrastructure; our Playwright versus Selenium comparison examines these criteria in more detail.
Introducing AI into test creation raises a question of trust. Solutions on the market promise fully autonomous testing, but none can guarantee that a generated test checks the exact behaviour it was meant to verify. A test that cannot be trusted is worse than having no test. There is little value in ending up with a large suite of passing tests when nobody can say whether they are reliable. The key question was therefore where human work and review should enter the process so that collaboration between people and AI would be as reliable and as fast as possible. In testing, reliability and accuracy must always take priority over speed.
A technology stack that makes human review easier
The target stack was Robot Framework with Browser Library, which controls the browser through Playwright. This combination has a characteristic that proved decisive when working with AI: Robot Framework keeps tests in a readable, almost sentence-like form, so a person can check what a test does without reading its implementation. For machine-written code, this is more than a cosmetic advantage. The faster a person can understand what a test verifies, the easier it is to review and the more work can reasonably be entrusted to AI.
The tests use the Page Object Model design pattern, which separates scenarios from selectors and page operations. This is an established approach rather than a custom invention. The creators of both Playwright and Selenium recommend it, and it has long been regarded as part of best practice.
How one test is created
The input is a test case described in sentences by someone who understands the required behaviour, rather than written as code. A recording of the journey and test data can be added where needed.
The input is first approved by a reviewer: someone who understands the requirement and can judge whether the finished test verifies the right thing. This is usually a tester, although it does not have to be. Only then does AI prepare a plan, open the application in a browser through Playwright MCP and locate the actual elements. It also determines whether existing methods in the page objects can be reused or new ones are needed. The result is a Robot Framework test together with any page objects it required. Having AI inspect the real application is the difference between a test that can run and one it has imagined based on how websites usually look.
Separate agent roles with their own workflows handle code preparation and review. Shared rules define the test structure, selector choice and use of the Page Object Model. Automated checks for syntax, duplication and linting run on top of this process.
Where it breaks down: a passing test that verifies nothing
The greatest risk with generated tests is not that they will fail to run. It is that they will not test what the scenario actually requires.
When AI is told to “make the test pass”, it has two options. It can write a test that verifies the required behaviour, or change the assertion so that it matches what the application currently does. The second option is easier and looks the same at first glance. The run is green.
This happened, for example, in a test intended to verify validation of a form field: entering an invalid value was supposed to display an error message. Instead, AI entered a valid value in the field. The form submitted, the test passed and everything looked correct at first glance. The one thing the test had been created to verify remained unchecked.
The rules must therefore prevent situations like this. They explicitly prohibit changing the meaning of a scenario, omitting required checks or adapting the expected result to the application’s current behaviour without human approval. A green run remains one piece of evidence for the review, rather than its conclusion.
What the reviewer evaluates
Before a new test or a repair is accepted, the review checks:
- whether the steps, test data and expected results match the written specification;
- whether the code contains assertions that actually evaluate the required outcome of the scenario;
- whether selectors and page operations are used correctly and the test remains understandable for future maintenance;
- whether a proposed repair preserves the original purpose of the test and all required assertions.
When a test fails
When a test fails, AI can reopen the application through Playwright MCP and propose a selector or page-object repair. This process is known as self-healing. The proposal goes through the same review as a new test: did it preserve the original purpose and all required assertions?
With a suite containing hundreds of tests, however, a different question is often more important: which failures are related? The solution therefore groups failures by the meaning of error messages, affected locators and similar symptoms. Instead of a list of red rows, the team can see that many of them share one cause and knows where to begin.
Tests can run locally or as part of continuous integration and deployment (CI/CD). Results are available in the standard Robot Framework reports and in Allure.
One repository for multiple projects
The projects live in a shared repository, following a monorepo structure. The shared part contains workflow skills, agent definitions, general rules and helper functions that do not depend on a specific project, covering everything from setting up a scenario and creating tests and page objects to analysing reports and making repairs.
In practical terms, a new project starts with procedures that have already proved effective elsewhere rather than an empty repository. Separate branches and isolated working copies of the same repository, known as worktrees, allow several people to work on the suite at the same time.
What changed as a result
- Creating one test changed from hours of work to reading and approving generated code in a fraction of that time.
- The Selenium migration could be completed at a scale that would have taken a very long time manually.
- Human attention moved to where it adds the most value: the specification and the question of whether the test verifies the right thing.
- Failures are handled in groups with a shared cause rather than one by one.
- A new project starts with established procedures, agents and rules rather than from scratch.