Evaluation of approximate comparison methods on Bloom filters for probabilistic linkage

Area of Science:

• Data Science
• Computer Science
• Information Security

Background:

• Privacy-preserving record linkage (PPRL) is crucial for secure data matching.
• Bloom filters are widely researched for privacy in data linkage.
• Limited studies explore Bloom filters within probabilistic frameworks for linkage quality.

Purpose of the Study:

• Evaluate approximate comparison methods for Bloom filters in probabilistic record linkage.
• Assess Sørensen-Dice coefficient, Jaccard similarity, and Hamming distance.
• Compare these methods against exact matching within the Fellegi-Sunter model.

Main Methods:

• Utilized synthetic datasets with simulated errors and a real-world health dataset.
• Estimated partial weight curves for similarity scores to partial weights.
• Conducted deduplication linkages using partial weight curves and compared with exact matching.

Main Results:

• Approximate comparisons significantly outperformed exact comparisons in linkage quality.
• Field-level partial weight curves yielded the highest quality results.
• Sørensen-Dice and Jaccard similarity demonstrated consistent performance across datasets.

Conclusions:

• Bloom filter similarity comparisons offer comparable quality to Jaro-Winkler with unencrypted linkages.
• Probabilistic record linkage benefits substantially from Bloom filter similarity comparisons.
• Partial weight curves provide the most effective approach for optimizing linkage quality.