Why the original solution was not enough
The client used a commercial tool for creating automated tests that promised automation without coding knowledge. This promise held true to a degree: a junior tester could create mostly simpler tests without coding. More complex scenarios, however, required developer capacity. Test automation therefore competed with product development for the same capacity.
The tool did not always generate stable selectors, which are the rules a test uses to locate elements in an application. Automatically generated selectors therefore still needed review and adjustment.
The objective was broader than a technical tool swap. The replacement needed to reduce dependency on the development team, expand coverage and preserve regression checks throughout the migration.
Migration with both suites running
The new framework was built from scratch using the open-source tools Selenium for web tests and Appium for mobile tests. Both parts used the same Page Object Model design pattern, separating test scenarios from selectors and operations on pages or screens. This kept the architecture consistent while using different technologies to interact with the web and mobile applications.
Selected high-value scenarios moved in stages while the original and replacement suites ran side by side for a period. The original tool was retired only after the replacement had been verified.
This approach limited the risk of losing important regression checks during the transition. It also avoided copying every historical test mechanically and focused the effort on scenarios that still provided value.
Operation, diagnostics and scope
The tests were integrated into GitLab CI/CD, supported on-demand runs and used parallel execution. Reports captured test steps, screenshots, the browser console and network traffic so failures could be investigated without lengthy reproduction.
Alongside reports, the solution included error pattern analysis to examine recurring errors. Its overview showed which parts of the test suite had the most failures. The team could see the most problematic areas at a glance and decide where to start investigating. A related article explains how similar grouping can be extended with AI-assisted test failure triage.
- hundreds of web and mobile tests;
- automated accessibility checks;
- production monitoring of several key journeys, running continuously from a Kubernetes pod.
What changed for the team
- An open-source solution fully replaced the original commercial tool.
- Developers were no longer needed to develop and maintain the automated tests.
- Tests could run in CI/CD or on demand, while parallel execution reduced the wait for results.
- Diagnostic reports helped investigate individual failures, while error pattern analysis highlighted the most problematic parts of the test suite.
- One solution covered web regression, mobile, accessibility and selected production journeys.