Density ratios to evaluate and improve the utility of synthetic data

Thom Benjamin Volker
t.b.volker@uu.nl

Imagine you have access to all the data in the world…

Imagine you have access to all the data in the world. What a privacy disaster would that be… But also, what a great opportunity to learn about the world! How many research questions could we answer if we had that much data. Unfortunately, or fortunately, we will never get access to all the data in the world, not in the last place because the data cannot be shared openly.

If real data is no option,

maybe synthetic data is!

Synthetic data is fake data, simulated data or generated data. As opposed to real data, synthetic data is not collected from real people, but generated from a model. If the model is a good approximation to reality, the synthetic data can be very useful. Because the synthetic data do not map back to individuals, it substantially reduces privacy risks.

Synthetic data

Fake data, generated data, simulated data, digital twins

Potential use-cases of synthetic data

• Advancing access to private data for research (e.g., in statistical institutes)

• Advancing open science workflows

• Educational materials

• Software / model testing

Synthetic data generation cycle

1. Create synthetic data with simple models

2. Evaluate the quality of the synthetic data

3. If necessary, add complexity (alter models, transformations, interactions)

4. Iterate between (2.) and (3.) until synthetic data is good enough

How do we know whether the synthetic data is good enough?

Intuitively

Can we use the synthetic data for the same purposes as we wanted to use the real data?

Do the observed and synthetic data have similar distributions?

Practically

Do the observed and synthetic data produce similar results under the same analysis?

Can we distinguish between the observed and synthetic data?

Density ratios for utility¹

r(x) = \frac{p(\boldsymbol{X}_{\text{syn }})}{p(\boldsymbol{X}_{\text{obs}})}

We propose that density ratios are very suitable for this task. Density ratio estimation is a set of techniques developed in machine learning to estimate the ratio of two probability density functions. The density ratio can be used for various tasks, as broad as change-point detection, prediction and two-sample testing. But it is also very useful for evaluating the utility of synthetic data. That is, if the observed and synthetic data have similar density over the entire multivariate space, then any analysis will yield similar results. So, this means that if the density ratio is close to one at every point in the multivariate space, then the synthetic data is good enough. If the density ratio is far from one in some subspace of the data, we have to improve the synthetic data in that subspace. Importantly, the density ratio is estimated directly, rather than estimating the two probability density functions separately and then taking their ratio to improve estimation accuracy.

1. See Masashi, Suzuki & Kanamori (2012). Density ratio estimation in machine learning.

Density ratios for utility evaluation

Density ratios in practice

1. Estimate the density ratio directly and non-parametrically

• Implemented in R-package densityratio

2. Calculate a discrepancy measure for the synthetic data

• Kullback-Leibler divergence; Pearson divergence

3. Compare discrepancy measures for different data sets

4. Optionally: Test the null hypothesis p(\boldsymbol{X}_{\text{syn}}) = p(\boldsymbol{X}_{\text{obs}})

Density ratios for synthetic data (multivariate examples)

U.S. Current Population Survey (n = 5000)¹

• Four continuous variables (age, income, social security payments, household taxes)
• Four categorical variables (sex, race, marital status, educational attainment)

Synthetic data models

(Multinomial) logistic regression for categorical variables

1. Linear regression
2. Linear regression with transformations (cubic root)
3. Linear regression with transformations and semi-continuous modelling

1. Thanks to Jörg Drechsler for sharing the data.

Utility of the synthetic data

Reweighting synthetic data: regression coefficients

Reweighting synthetic data: regression coefficients

Reweighting synthetic data: regression coefficients

	Observed	Synthetic	Reweighted
b0	-0.0179771	0.0203401	-0.0189604
b1	0.5152371	0.0385202	0.4779748

Other advantages of density ratios for utility

Use density ratios to discard synthetic outliers

High-dimensional extensions

• Find a m < p-dimensional subspace in which the synthetic and observed data are maximally different

• Estimate the density ratio in this subspace

Automatic cross-validation for hyperparameter selection

Thanks for your attention!

Even if it was simulated…

Questions?

t.b.volker@uu.nl