Probabilistic master lists: integration of patient records from different databases when unique patient identifier is...

Area of Science:

• Computational statistics
• Data integration
• Health informatics

Background:

• Integrating datasets is challenging when unique identifiers are absent.
• Existing methods may struggle with imperfect or non-standard data fields.
• Accurate client re-identification is crucial for data linkage and analysis.

Purpose of the Study:

• To develop and validate a Bayesian probability model for merging databases without unique client identifiers.
• To assess the model's accuracy using imperfect and overlapping client identifiers.
• To demonstrate the model's effectiveness on real-world health expenditure data.

Main Methods:

• Utilized Bayesian probability models to estimate the likelihood of record linkage.
• Employed a set of imperfect identifiers (e.g., diagnosis, name) to assess client identity.
• Accounted for inter-dependencies among identifiers, allowing for overlapping and redundant data.

Main Results:

• The algorithm achieved 100% correct classification of new and known cases when using 12 identifier fields.
• Accuracy decreased with smaller training datasets (<100 records) or fewer identifier fields (<7).
• Performance exceeded 90% accuracy when the testing to training data ratio surpassed 4:1, improving with higher ratios.

Conclusions:

• The developed automated and mathematical procedure accurately merges data from disparate sources lacking unique identifiers.
• The Bayesian approach effectively leverages imperfect and overlapping clues for robust case re-identification.
• The model shows significant promise for data integration in scenarios with incomplete or non-standard identifying information.