Record linkage in public health datasets: a practical experience in a fast in-process analytical database

Area of Science:

• Public Health Informatics
• Database Systems
• Epidemiological Surveillance

Background:

• Accurate record linkage is vital for public health surveillance.
• Integrating data from systems like SIM and SIVEP-Gripe presents challenges.

Purpose of the Study:

• To evaluate a novel algorithm for linking Mortality Information System (SIM) and Influenza Epidemiological Surveillance Information System (SIVEP-Gripe) records.
• To assess the performance of this algorithm implemented in DuckDB.

Main Methods:

• A hybrid deterministic-probabilistic approach was employed.
• Similarity metrics including Jaro and Jaro-Winkler were utilized.
• The algorithm was compared against a previously validated method across various prevalence scenarios.

Main Results:

• The DuckDB-based algorithm demonstrated significantly faster processing, up to 100x improvement.
• High sensitivity, specificity, and predictive values were maintained.
• The solution proved scalable for large-scale, real-time applications.

Conclusions:

• DuckDB offers high-performance data integration capabilities for complex tasks.
• The algorithm is suitable for resource-limited public health settings.
• Efficient record linkage is essential for timely and accurate public health insights.