XenonStack BlogSynthetic Data Generation: Methods, Privacy and Risks

Synthetic data is artificially generated data that reproduces the statistical patterns of a real dataset without copying its records. Enterprises use synthetic data generation to fill test environments, train and evaluate machine learning models, share data across teams and partners, and simulate rare events that real data barely contains. Methods range from simple rules to statistical models, generative adversarial networks (GANs), diffusion models and large language models (LLMs) that write synthetic text.

Synthetic data is not automatically anonymous and not automatically safe to train on. European regulators assess it with the same identifiability test as any other data, published research shows generators can leak training records, and models trained recursively on their own output can lose the rare cases that matter most. Treat it as an engineered data product with fidelity, utility and privacy tests, not as a compliance shortcut.

What is synthetic data?

A synthetic dataset is produced by a process, usually a model fitted to real data, that samples new records with similar distributions, correlations and structure. Done well, a query, chart or model trained on the synthetic version gives answers close to those from the original. NIST lists "generating synthetic data using models" among de-identification approaches in SP 800-188, De-Identifying Government Datasets (September 2023), alongside traditional techniques such as suppression and generalisation.

Three forms are common in practice:

• Fully synthetic: every record is generated; no original row is released.
• Partially synthetic: only sensitive columns are replaced with generated values, and the rest stay real.
• Hybrid or augmented: real data is kept and synthetic records are added, for example to balance a rare class.

Partially synthetic and hybrid datasets keep real values, so they carry more disclosure risk than fully synthetic data and should be governed as personal data unless an assessment shows otherwise.

How does generative AI create synthetic data, and which method fits?

The right generator depends on the data type, how much realism you need and how much privacy risk you can accept. The table summarises the main families.

Some evidence behind the table: Xu et al. showed a conditional GAN for tabular data handling mixed discrete and continuous columns, multimodal values and imbalanced categories, and outperforming Bayesian network baselines on most real datasets in their benchmark (Modeling Tabular Data using Conditional GAN, 2019). Ho, Jain and Abbeel's denoising diffusion probabilistic models (2020) reported an FID of 3.17 on unconditional CIFAR10. For text, Self-Instruct (Wang et al., 2022) had GPT-3 generate its own instructions, filtered invalid and near-duplicate examples, and reported a 33 percentage point improvement on Super-NaturalInstructions, close to InstructGPT-001. For the internals of adversarial training and its variants, see our guide to GAN architecture, and for model families more broadly, generative AI models and their evaluation.

Where does synthetic data deliver value in the enterprise?

Synthetic data complements rather than replaces good source data. Its quality ceiling is set by the real data the generator learns from, so data quality management on the source still matters.

How do you evaluate synthetic data quality and privacy?

Every synthetic dataset should pass three tests before release: fidelity, utility and privacy. They pull in different directions, so set thresholds per use case.

Utility: train on synthetic, test on real

The TSTR approach, described by Esteban, Hyland and Rätsch for medical time series (2017), trains a model on synthetic data and measures it on a real hold-out set. It answers the question decision makers care about: would a model built on this data work in production?

Privacy: membership inference and the privacy-utility trade-off

Shokri et al. showed that, with only black-box access, an attacker can often tell whether a specific record was in a model's training set (Membership Inference Attacks against Machine Learning Models). Generators are models too. Xie et al. noted that GAN distributions can concentrate around training points, a privacy risk for data such as medical records, and proposed adding noise to gradients during training (DPGAN, 2018).

Stadler, Oprisanu and Troncoso's quantitative evaluation (Synthetic Data: Anonymisation Groundhog Day) found that across state-of-the-art generators, synthetic data either failed to stop inference attacks or failed to retain utility, and did not give a better privacy-utility trade-off than traditional anonymisation. They also found the trade-off hard to predict in advance. That is why privacy must be measured on each release, not assumed.

Differential privacy

Differential privacy is, in NIST's words, "a mathematical framework that quantifies privacy loss to entities when their data appears in a dataset". Training a generator with differential privacy bounds how much any one record can influence the output. NIST's SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees (March 2025) sets out a "differential privacy pyramid" of factors and common implementation pitfalls. Ask vendors for the privacy parameters, what unit is protected (a row, a person, a household) and how the implementation was verified.

Is synthetic data anonymous under GDPR, and what does the EU AI Act require?

GDPR and UK GDPR

Under the GDPR, information falls outside data protection law only if the people it relates to are not identifiable using means reasonably likely to be used (Recital 26). The European Data Protection Board's Guidelines 02/2026 on Anonymisation, adopted on 7 July 2026 for public consultation, build their assessment on that test. They name synthetic data directly: inference-based risks "would, in particular, be the case for AI models or synthetic data", and a meaningful inference about a person breaches their "No Inference" criterion. They also note that it may be possible to infer that an individual's data was included in the original dataset.

The EDPB's earlier Opinion 28/2024 on AI models (December 2024) concluded that models trained on personal data "cannot, in all cases, be considered anonymous"; the likelihood of obtaining personal data from queries "should be insignificant", assessed case by case. It lists attribute and membership inference among attacks controllers should test against, and cites differential privacy as an example of a privacy-preserving technique. The same logic applies to a generator and the data it produces.

In the UK, the ICO's anonymisation guidance covers identifiability and the "motivated intruder" test; the ICO states the guidance is under review following the Data (Use and Access) Act, which became law on 19 June 2025. Training a generator on personal data is itself processing, so the lawful basis, purpose and security controls apply to that step even if the output is later judged anonymous.

EU AI Act Article 10

For high-risk AI systems, Article 10 of Regulation (EU) 2024/1689 requires training, validation and testing data to be "relevant, sufficiently representative" and, as far as possible, free of errors and complete. Data governance practices must cover design choices, data origin, preparation such as labelling and cleaning, assumptions, bias examination and mitigation, and data gaps. Synthetic records used in training fall under these duties like any other data, so document how they were generated and how representativeness was checked.

Article 10(5) allows special categories of personal data to be processed for bias detection and correction only under conditions, the first being that this "cannot be effectively fulfilled by processing other data, including synthetic or anonymised data". In other words, the Act expects providers to try synthetic or anonymised data first. The AI Act Explorer shows pending amendments to Article 10, including to paragraph 5, so check the consolidated text before relying on it. Build these checks into your wider data governance programme.

What is model collapse, and how do you avoid it?

Model collapse is the degradation that occurs when models are trained on data produced by earlier models. Shumailov et al. reported in Nature (vol. 631, July 2024), "AI models collapse when trained on recursively generated data" (first released as The Curse of Recursion, 2023), that indiscriminate use of model-generated content causes irreversible defects in which the tails of the original distribution disappear. They observed it in LLMs, variational autoencoders and Gaussian mixture models.

The tails are exactly where fraud, failures and minority groups live. A follow-up study by Gerstgrasser et al. (Is Model Collapse Inevitable?, 2024) found that replacing real data with each generation's synthetic output tends towards collapse, while accumulating synthetic data alongside the original real data avoided it across the model sizes and architectures they tested. Practical rules follow:

• Keep a versioned, real reference dataset and never discard it in favour of synthetic data.
• Tag synthetic records with provenance so they are not unknowingly fed back into later training runs.
• Monitor tail metrics, such as rare-class recall, not just averages.
• Cap the synthetic share of training data per use case and justify it with TSTR results.

How should an enterprise adopt synthetic data?

1. Choose one use case with a measurable outcome. Test data for a regulated application or augmentation for one imbalanced model is a better start than a company-wide platform.
2. Classify the source data and legal position. Confirm the lawful basis for training the generator, and decide whether the output will be treated as personal data or assessed for anonymity.
3. Select the generator by data type. Use the method table above; prefer the simplest method that meets utility targets.
4. Set acceptance thresholds. Agree fidelity, TSTR utility and privacy-attack thresholds with data owners and the DPO before generating.
5. Run privacy attacks and document results. Membership and attribute inference tests, plus differential privacy where risk is high.
6. Record lineage. Store generator version, training data snapshot, parameters and evaluation reports with each dataset, which also supports EU AI Act Article 10 documentation.
7. Monitor in production. Track drift between synthetic and real data and retire datasets that no longer match reality.

If you need support building and validating models on real and synthetic data, see our machine learning development services.

What are the trade-offs, and when should you not use synthetic data?

Frequently asked questions

What is synthetic data in simple terms?

It is generated data that mimics the statistical patterns of real data without reproducing its records, used for testing, model training, data sharing and simulating rare events.

Is synthetic data automatically GDPR compliant?

No. The EDPB's 2026 draft anonymisation guidelines apply the same identifiability and inference analysis to synthetic data as to other data, and training the generator on personal data is itself processing.

Which method is best for synthetic data generation?

It depends on the data. Rule-based and statistical methods suit testing and simple tables; conditional GANs suit mixed-type tabular data; diffusion models suit images and audio; LLMs suit text, instructions and test dialogues.

Can synthetic data be used for AI training?

Yes, for augmentation and fine-tuning, provided you measure utility on real data and keep real data in the training mix. Research on model collapse shows that replacing real data with recursively generated data degrades models, while accumulating alongside real data avoided collapse in tested settings.

How does the EU AI Act treat synthetic data?

Article 10 data quality and governance duties apply to all training, validation and testing data for high-risk systems, and Article 10(5) requires providers to consider synthetic or anonymised data before processing special categories of personal data for bias correction.