Estimating parameters for probabilistic linkage of privacy-preserved datasets

Area of Science:

• Data science
• Computer science
• Statistics

Background:

• Probabilistic record linkage is crucial for data integration but challenging with privacy-preserved datasets.
• Traditional methods rely on data quality assessments and manual inspection, which are not feasible with encrypted data.
• Bloom filters offer a privacy-preserving mechanism for data linkage.

Purpose of the Study:

• To present a method for estimating probabilities and thresholds for probabilistic privacy-preserved record linkage using Bloom filters.
• To evaluate the accuracy and applicability of the proposed method on synthetic and real-world data.

Main Methods:

• A simulation study with synthetic data (0-20% error) and real-world administrative data was used for testing.
• Datasets were privacy-preserved using Bloom filters for each field.
• The expectation-maximization (EM) algorithm was adapted to estimate match probabilities and determine optimal threshold values.

Main Results:

• The proposed method achieved F-measure comparable to calculated probabilities on synthetic data, even with high error rates.
• On real-world administrative data, the method yielded a higher F-measure than traditional calculated probabilities.
• Threshold estimation provided results close to the optimal F-measure.

Conclusions:

• The developed method is highly accurate for probabilistic linkage of privacy-preserved datasets, even with significant data errors.
• This approach represents a significant advancement towards a complete operational framework for privacy-preserving data linkage.