A scalable and transparent data pipeline for AI-enabled health data ecosystems

Area of Science:

• Artificial Intelligence
• Health Informatics
• Data Science

Background:

• Lack of transparency in AI data preparation hinders reproducibility, debugging, bias detection, and regulatory compliance.
• Manual data extraction processes are error-prone and inefficient for developing reliable AI systems.
• Trustworthy AI requires robust methods for data preparation and traceability.

Purpose of the Study:

• To introduce a formal data preparation pipeline specification for AI applications.
• To improve transparency and traceability in the AI data extraction process.
• To address challenges in developing reliable and reproducible AI models from health data.

Main Methods:

• Developed a declarative language for extracting AI-ready datasets from health data using HL7 Fast Healthcare Interoperability Resources (FHIR) profiles.
• Created a common data model tailored for AI use cases, defining phenotypes and AI feature specifications.
• Implemented a scalable feature repository for executing data preparation pipelines, converting complex EHR data into a structured format.

Main Results:

• Successfully deployed and tested the data preparation pipeline in three research projects, including predicting complications after cardiac surgery.
• The implemented software ensures reliable, fault-tolerant processing, producing AI-ready datasets with comprehensive metadata.
• The framework demonstrated flexibility in defining diverse features with specific temporal and contextual criteria across pilot use cases.

Conclusions:

• The proposed formal data preparation pipeline enhances transparency and traceability in AI development.
• The framework supports the creation of reliable AI-ready datasets from complex health data.
• This methodology facilitates the development of trustworthy AI systems in healthcare and other data-intensive domains.