An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP...

Area of Science:

• Medical Informatics
• Health Data Management
• Clinical Trial Operations

Background:

• The Medical Informatics in Research and Care in University Medicine (MIRACUM) consortium developed an IT system for clinical trial recruitment support.
• This system utilizes the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) populated with German Fast Healthcare Interoperability Resources (FHIR) via an Extract-Transform-Load (ETL) process.
• The current bulk ETL process is computationally intensive and inefficient for daily data updates required for recruitment.

Purpose of the Study:

• To enhance the existing ETL process by incorporating incremental loading capabilities.
• To enable efficient processing of daily updated data for improved clinical trial recruitment support.

Main Methods:

• Analysis of requirements for incremental loading based on the existing bulk ETL process.
• Literature review to identify suitable adaptable approaches for incremental data integration.
• Implementation of three distinct methods for integrating incremental loading into the ETL workflow.
• Development of a test suite to evaluate data correctness and performance of incremental loading against bulk loading.

Main Results:

• The developed ETL process successfully supports both bulk and incremental loading.
• Incremental loading demonstrated an 87.5% reduction in execution time compared to bulk loading for daily data changes (2.12 min vs. 17.07 min).
• No discrepancies in data were observed in the OMOP CDM between the incremental and bulk loading methods.

Conclusions:

• Incremental loading offers a more efficient alternative to daily bulk loads for updating clinical trial recruitment data.
• Both loading methods yield identical data in the OMOP CDM, ensuring data integrity.
• The developed incremental ETL logic is internationally applicable, not limited by German FHIR profiles, and recommended for daily updates after an initial bulk load.