Enabling Agile Clinical and Translational Data Warehousing: Platform Development and Evaluation.

Area of Science:

• Biomedical Informatics
• Data Science in Healthcare
• Clinical Data Warehousing

Background:

• Data-driven medical research relies on robust infrastructure for insights and decision support.
• Clinical and translational data warehouses (e.g., i2b2, tranSMART) unify heterogeneous datasets for research use cases.
• Challenges include platform complexity, time-consuming data loading, and the need for domain expertise.

Purpose of the Study:

• To develop a platform addressing challenges in managing clinical and translational data warehouses.
• To enhance agility in platform management and data loading processes.
• To facilitate closer collaboration between informaticians and clinical researchers.

Main Methods:

• Formulated system requirements for agile platform management and data loading.
• Designed a cloud infrastructure with unified interfaces for multiple warehouse platforms.
• Implemented a declarative, meta-loading data pipeline automating data restructuring and cleansing.

Main Results:

• The platform supports i2b2 and tranSMART with integrated security.
• The data-loading pipeline successfully processed previously un-loadable data without extensive preprocessing.
• Configuration file sizes were reduced by up to 22x for tranSMART and 1135x for i2b2, significantly lowering effort.

Conclusions:

• The developed platform substantially reduces effort in managing clinical data warehouses and loading complex data formats.
• Compact configuration files facilitate iterative refinement of data representations.
• The open-source platform offers a cloud-based solution for the research community.