AI-assisted testing

AI-generated test data: synthetic data instead of a production copy

Test automation needs users, orders, addresses, documents and unusual combinations of states. The quickest shortcut is often a copy of the production database. That copy also brings personal data, access risks, stale records and data that was not designed for a particular test objective.

Synthetic data is created artificially from rules or learned patterns. AI can help design and generate it, but the label “synthetic” does not automatically mean anonymous, safe or useful. A sound solution combines a data contract, controlled generation and measurable checks of privacy and test value.

Why copying and renaming production is not enough

A production copy feels realistic because it contains genuine relationships and distributions. It also expands the number of locations where information must be protected. A test environment may have weaker access controls, longer retention, verbose logs or integrations that unexpectedly send a message to a real customer.

Replacing a name with Test User does not solve the problem. An email address, phone number, order identifier, free text or combination of age, town and date may still allow a record to be linked to a person. Pseudonymised information remains personal data where additional information can reconnect it to an individual.

Article 5 of the GDPR includes purpose limitation and data minimisation principles. In practice, they support a useful question: does this test truly need production personal data, or is a record with the right structure and properties sufficient? This is not an individual legal assessment; the applicable basis, exceptions and safeguards should be considered with the organisation’s data-protection function.

The broader lifecycle, masking and access-control questions are covered in test data and GDPR. Synthetic generation is one option within that strategy, not a replacement for it.

Synthetic, masked and anonymised are not the same

Three different approaches are often mixed together:

• Masking or pseudonymisation changes identifiers in existing records. It preserves many real relationships, but linking data back to a person may remain possible.
• Anonymisation aims to reduce identifiability so that information is no longer personal data. Assessment depends on the method, auxiliary information available and realistic re-identification risk.
• Synthetic generation creates new records. It can be purely rule-based, or a model can imitate statistical properties of source data.

A generative model trained on sensitive records may reproduce exceptional patterns or parts of its input. “AI-generated” is therefore not a synonym for “anonymous”. The EDPB material on anonymisation and pseudonymisation emphasises identifiability in context. If the legal status of anonymisation matters, a tool’s marketing label is not enough; the method and risk assessment need to be documented.

When AI adds value to test data

For many tests, simple rule-based generation is best. A library can produce a valid email, date or random identifier quickly, cheaply and reproducibly. AI is most useful where data has richer content or many meaningful combinations need to be proposed.

Suitable examples include:

• complaint text with different lengths, tones and missing details;
• names and addresses following several countries’ formats without using real people;
• documents with varied layouts for OCR or classification;
• catalogue descriptions, search phrases and spelling mistakes;
• proposed boundary combinations of order, payment and delivery states;
• additional rare classes for model testing, provided bias is also assessed.

AI is not required for every database row. Stable identifiers, referential relationships, amounts and business rules are usually safer in a deterministic generator. A model can prepare content fields or propose cases; code then enforces the schema and invariants.

Start with the test objective, not the model

“How many synthetic customers should we create?” is too early a question. First name the behaviour to verify. Checkout might need an ordinary order, the last item in stock, a rejected payment, a discount combined with free shipping, and resumption after an interruption.

For each data profile, define:

1. Schema: required fields, types, formats and allowed values.
2. Relationships: the customer owns the order, line items sum to the total, and payment status permits the order state.
3. Invariants: rules that must always hold, such as a unique identifier or non-negative quantity.
4. Boundaries: minimum and maximum lengths, Unicode, leap day, an empty optional value or a very large basket.
5. Forbidden values: real corporate domains, valid phone ranges, production IDs or text containing a secret.
6. Expected outcome: which journey the data activates and how the test will judge it.

This data contract matters more than the prompt. Without it, a model can create convincing rows that violate subtle business rules.

A controlled generation process

A practical flow can use the following stages.

1. Minimise source context

Do not send a sample production record to an external model merely to explain a format. Use an empty schema, artificial examples, enums and a written description of the rules. Even when the organisation runs a model in its own controlled environment, purpose, access, retention and logging still need to be defined.

Sensitive data can enter not only a prompt but also attachments, a vector database, telemetry or diagnostic logs. A service assessment therefore needs the entire data flow, not only a statement that customer data is not used for training. The article on AI testing and data privacy covers these questions in more detail.

2. Generate named profiles

Instead of one request for ten thousand “realistic” rows, create named profiles: ordinary user, new account with no history, customer with several addresses, rejected payment and boundary-length text. A profile can be linked to a test and its frequency controlled.

AI can propose text values or combinations. A deterministic layer adds keys, timestamps, totals and referential integrity. For repeatable tests, store the random seed and versions of the generator, model, prompt and rules so a failure can be reproduced.

3. Validate before use

A generated file should not go directly into tests. Automated validation should check:

• schema, types, required values and uniqueness;
• referential integrity and business invariants;
• allowed ranges and the representation of profiles;
• absence of forbidden domains, identifiers and secret patterns;
• similarity to source records if a model learned from real data;
• the expected outcome against a small reference set.

The NIST Synthetic Data Report Tool treats synthetic data through two distinct questions: utility and privacy. This is a useful design principle beyond that tool. Data can be safe but useless, or realistic while remaining too similar to a sensitive source.

4. Publish a version, not a random result

An approved dataset should have a version, owner, creation date, purpose and known limitations. A regression test should not call a model without controls and change its own input on every run. Dynamic generation is suitable for fuzzing or discovery, whereas regression requires the exact case to be reproducible.

A good combination is a stable baseline plus a controlled batch of new data. When a new case reveals a defect, its smallest reproducible form becomes part of the regression set.

How to verify utility

“It looks realistic” is not a test criterion. Measure utility against the objective:

• coverage of enums, states and pairs of dependent values;
• the number of boundary and negative cases;
• distributions of text lengths, amounts and line-item counts;
• ability to activate the expected application branches;
• agreement with an independent control calculation;
• successful data loading without manual repairs.

For analytics or machine-learning tests, statistical distributions and model performance may need comparison across real and synthetic validation data. Similarity is not an unlimited goal: an overly faithful copy of an exceptional record can increase disclosure risk.

End-to-end tests often need only a smaller, deliberately designed set. The guide to test data for E2E tests covers state creation, cleanup and scenario isolation.

Limitations and common mistakes

Synthetic data does not remove every need to check real behaviour. It may omit a rare combination, preserve bias from a source or create a world that is too clean compared with the messy inputs integrations receive. A production decision may still require a controlled test with minimised real data under an appropriate legal and security regime.

Another mistake is treating a public chatbot as an improvised anonymisation service. A model should not be the final authority deciding whether text contains personal data. Technical rules, control samples and expert assessment are necessary where the risk warrants them.

Generation also has costs: model operation, validation, versioning and rule maintenance. If a simple function can reliably create ten rows, AI only adds complexity.

What the team gains

Controlled synthetic data reduces the need to distribute production copies, speeds up preparation of particular states and deliberately creates cases that occur rarely in a production sample. Tests become more reproducible because input comes from versioned rules rather than a random database snapshot.

The greatest value appears when every profile has a clear purpose. The team knows why a record exists, which scenario it supports and which input defect validation should stop. The synthetic set becomes a managed part of test architecture rather than another anonymous file on a shared drive.

Next step

Choose one scenario that currently requires a production export. Define its minimum schema, relationships, five boundary profiles and forbidden values. Generate a small set without sending a real record to the model, validate it automatically and compare whether it activates the same application branches. Only then decide whether AI adds enough value over a rule-based generator.