A late-binding, distributed, NoSQL warehouse for integrating patient data from clinical trials

Area of Science:

• Biomedical Informatics
• Clinical Data Management
• Health Data Science

Background:

• Clinical trial data are fragmented across multiple vendor systems with varying standards.
• Integrating and standardizing this data is crucial for monitoring and reporting but presents technical challenges.
• Existing systems struggle with large volumes of inconsistent data and evolving requirements.

Purpose of the Study:

• To develop a data warehouse solution for a comprehensive clinical data repository.
• To integrate, standardize, and provide timely access to patient data from any source.
• To support diverse analytic tasks for clinical studies.

Main Methods:

• Implemented a data warehouse using Apache HBase (NoSQL column store) and Apache Phoenix (relational query engine).
• Utilized an extract-load-transform (ELT) pattern, deferring data mapping until query time.
• Separately versioned data and transformations to enable dynamic data representation and historical analysis.

Main Results:

• The solution efficiently loads large data volumes under ambiguous specifications.
• Maintains a single data copy, transforming it dynamically without extra storage.
• Allows concurrent multiple data representations and simplifies data cut maintenance for adaptive trials.

Conclusions:

• The developed data warehouse offers a scalable, secure, and redundant solution.
• Combines NoSQL flexibility with relational query engine robustness.
• Effectively supports a wide range of applications including data management, medical review, and safety signal detection.