When to conduct probabilistic linkage vs. deterministic linkage? A simulation study.

Area of Science:

• Data Science
• Biostatistics
• Health Informatics

Background:

• Effective record linkage is crucial when unique identifiers are absent, relying heavily on data quality and available variables.
• Previous studies on probabilistic versus deterministic linkage yielded inconclusive results due to variations in data quality, methodology, and validation.

Purpose of the Study:

• To investigate how data characteristics influence the performance of probabilistic and deterministic record linkage methods.
• To compare the effectiveness and efficiency of probabilistic and deterministic linkage strategies under various simulated data conditions.

Main Methods:

• Developed 96 simulated scenarios with non-unique identifiers, varying discriminative power, missing/error rates, and file sizes.
• Assessed linkage method performance using standard validity metrics (e.g., PPV, sensitivity) and computation time.
• Systematically introduced data imperfections to mimic real-world record linkage challenges.

Main Results:

• Deterministic linkage demonstrated higher Positive Predictive Value (PPV), while probabilistic linkage excelled in sensitivity.
• Probabilistic linkage consistently provided a superior balance between sensitivity and PPV across diverse data quality levels.
• Deterministic linkage was computationally faster (<1 min) than probabilistic linkage (2 min to 2 hours), with comparable performance in very low error (<5%) scenarios.

Conclusions:

• The rate of missing data and errors in linkage variables is a critical factor in selecting between probabilistic and deterministic methods.
• Probabilistic linkage is generally recommended for its robustness, particularly with imperfect data.
• Deterministic linkage offers a more resource-efficient alternative when data quality is exceptionally high (less than 5% error).