AI wrote a test that passed but checked nothing
The test passes. The pipeline is green. The suite has thirty more tests than it did last week. One month later, a defect in the ordering process reaches production, even though the report says that process is covered by a test. The test merely opened the page, clicked a button and finished. It checked nothing. It passed because nothing could fail.
Why such a test is created
A language model composes a draft from the instructions and available context. If the instructions describe only the steps and do not specify the correct result, the model may create a script that clicks through the process without checking its substance. The same mistake can also occur in a manually written test.
It appears in several forms:
-
The test finishes without an assertion. It performs the scenario and asserts nothing. Green means only that the steps did not stop because of a technical error.
-
The test checks a side effect. It checks a heading or page address even though the purpose of the scenario is the creation of an order with the correct amount.
-
The test checks only its own input. It enters a value in a field and checks that the value appears there, but does not verify that the application processed the data.
-
The assertion is too general. Instead of checking a specific order, it looks only for the word “thank you”, which may appear on the page even after an unsuccessful result.
-
An error is suppressed. An exception is caught without being rethrown, and the test continues as if the step had succeeded.
Why this kind of test is particularly risky
A visibly failed test signals that the application, data or test itself needs to be investigated.
A test that checks nothing does the opposite. It looks like a success. It inflates the number of tests, increases the reported coverage and creates the belief that the journey is protected. The team may rely on the green result even though the test has not confirmed the essential behaviour.
Unlike a failing test, a weak assertion does not draw attention to itself. Without a deliberate review, it may remain in the suite for a long time.
How we address it
In AI-assisted testing, human review is therefore part of test creation, not a formal step at the end.
During the review, we focus particularly on the following questions:
-
Does the test check the objective of the scenario? For an order, this may mean the correct status, amount and items, not merely the display of a new page.
-
Can the test fail for the right reason? Depending on the risk, we verify a negative state or deliberately change the expected result. If the test passes even when the result is wrong, its assertion is insufficient.
-
Does the code suppress errors? We remove unjustified exception handling and fixed waits that merely hide a problem.
Only a draft with a clear expected result, suitable data and an understandable reason for any failure is added to the suite. AI prepares the draft; a person approves it.
What you gain
- Tests that check a result important to the user or the business.
- A lower risk of false confidence from a green but weak test.
- A clearer review process for drafts created by both people and AI.
Next step
Choose several critical tests and, for each one, name a specific defect that must make it fail. If you need an independent assessment of an existing suite, use a no-obligation consultation; we follow a similar approach when repairing a test suite.