An open-source probabilistic record linkage process for records with family-level information: Simulation study and...

Area of Science:

• Data Science
• Administrative Data Research
• Machine Learning Applications

Background:

• Administrative records often lack comprehensive information for policy analysis.
• Record linkage is crucial for integrating data from disparate sources by matching individuals.
• Existing record linkage solutions require evaluation for complex family-based datasets.

Purpose of the Study:

• To introduce and evaluate a novel machine learning approach, probabilistic record linkage for families (PRLF), using the Python RecordLinkage package.
• To assess the accuracy of PRLF under varying conditions of data degradation and match percentages.
• To compare the performance of PRLF against established record linkage tools (ChoiceMaker, Link Plus) in the context of regression modeling.

Main Methods:

• Developed a simulation of administrative records to test PRLF accuracy, manipulating match and data degradation rates.
• Applied machine learning algorithms for probability scoring within the PRLF framework.
• Compared regression model estimate performance using data linked by PRLF, ChoiceMaker, and Link Plus.

Main Results:

• PRLF accuracy was significantly influenced by data degradation (missing or mismatched fields) rather than the percentage of matches.
• Optimized implementations of PRLF, ChoiceMaker, and Link Plus produced comparable results for regression modeling.
• Ensemble methods were identified as a strength for improving match accuracy in record linkage.

Conclusions:

• PRLF offers a viable machine learning-based solution for family-based record linkage with administrative data.
• Data quality and degradation are critical factors affecting the accuracy of any record linkage method.
• Researchers can achieve similar analytical outcomes using different optimized record linkage tools for administrative datasets.