Analysis of a probabilistic record linkage technique without human review

Area of Science:

• Health Informatics
• Data Science
• Biostatistics

Background:

• Deterministic record linkage algorithms achieve high specificity but may miss true matches.
• Probabilistic linkage methods offer higher sensitivity but typically require manual review, increasing costs and time.
• Existing methods face challenges in balancing accuracy and efficiency, especially in large-scale data integration.

Purpose of the Study:

• To develop and evaluate an unsupervised probabilistic record linkage method to avoid human intervention.
• To improve upon the performance of a previously developed deterministic record linkage algorithm.
• To establish a reliable true-link threshold using the Expectation Maximization (EM) algorithm.

Main Methods:

• Utilized an Expectation Maximization (EM) algorithm as an estimator function for unsupervised probabilistic record linkage.
• Compared the EM algorithm's results against a manually reviewed gold-standard across two hospital registries.
• Evaluated performance metrics including sensitivity and specificity against a prior deterministic linkage algorithm.

Main Results:

• The EM algorithm demonstrated improved performance over the deterministic approach in record linkage accuracy.
• Achieved high actual specificities (99.43% and 99.42%) and sensitivities (99.19% and 98.99%) at estimated thresholds.
• The algorithm accurately estimated linkage parameters, providing reliable results without manual review.

Conclusions:

• The unsupervised probabilistic record linkage method using the EM algorithm is effective and accurate.
• This methodology offers a practical solution for record linkage when human intervention is not feasible.
• The approach enhances data integration capabilities in healthcare and other data-intensive fields.