SCALPEL3: A scalable open-source library for healthcare claims databases

Area of Science:

• Health Informatics
• Computational Biology
• Data Science

Background:

• Large observational databases (LODs) are crucial for health research but present scalability challenges.
• Existing infrastructures for analyzing health claims data can be slow and complex.

Purpose of the Study:

• Introduce SCALPEL3, an open-source framework designed for scalable analysis of LODs.
• Improve efficiency, reproducibility, and maintainability of health studies on large datasets.

Main Methods:

• SCALPEL3 leverages distributed computing, data denormalization, and columnar storage, inspired by web analytics.
• Performance was evaluated against the SAS-Oracle SNDS infrastructure using a 3-year healthcare claims dataset from 13.7 million patients.

Main Results:

• SCALPEL3 demonstrates superior horizontal scalability for handling large tasks compared to existing infrastructure.
• It offers comparable performance with fewer computing resources and provides interactive control for reproducible data processing.

Conclusions:

• SCALPEL3 significantly simplifies and enhances the scalability of studies using the French SNDS database.
• The framework is adopted by key health agencies in France, indicating its practical utility and impact.