Predicting Matching Quality of Record Linkage Algorithms on Growing Data Sets

Area of Science:

• Bioinformatics
• Data Science
• Population Health

Background:

• Accurate record linkage is crucial for identifying unique individuals in large datasets.
• Existing record linkage tools face challenges with limited attribute sets and scalability.
• The E-PIX matching tool's performance with minimal data (name, DOB, sex) requires evaluation.

Purpose of the Study:

• To analyze the matching behavior of the E-PIX tool using a restricted set of personal attributes.
• To develop a predictive model for estimating clerical review workload in record linkage.
• To identify optimal parameter sets for scalable and accurate duplicate detection.

Main Methods:

• Utilized a benchmark dataset of nearly 37,000 records from the Popgen biobank.
• Developed a predictive model to estimate workload for datasets scaled up to 10 times larger.
• Evaluated two parameter sets for comparable true duplicate detection rates and scalability.

Main Results:

• The developed model accurately predicts clerical review workload for scaled datasets.
• Identified parameter sets demonstrating comparable true duplicate detection but differing scalability.
• Demonstrated that unreviewed merging leads to homonym errors in large datasets (200,000 records).

Conclusions:

• The predictive model aids in optimizing record linkage parameters for improved data quality and efficiency.
• Scalable parameter sets are essential for managing large datasets and minimizing review workload.
• Careful parameter selection and review are necessary to prevent errors like homonym misidentification.