Multi-modal dataset creation for federated learning with DICOM-structured reports.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Data Science

Background:

• Federated learning faces challenges with heterogeneous medical datasets, including varied data formats, naming, annotations, and quality.
• Multi-modal learning paradigms require robust dataset harmonization, uniform data representation, and effective filtering for successful implementation.

Purpose of the Study:

• To develop an open platform for data integration and interactive filtering to simplify the creation of consistent multi-modal patient cohorts across multiple sites.
• To extend prior work by demonstrating applicability to diverse data types and streamlining datasets for federated training.

Main Methods:

• Utilized DICOM-structured reports for standardized linkage of imaging and non-imaging data within Python deep learning pipelines.
• Developed an open platform with interactive filtering capabilities for multi-site data integration and cohort creation.
• Applied the platform to harmonize imaging (CT), waveform (ECG) data, annotations (segmentations, pointsets), and metadata for a federated learning consortium.

Main Results:

• Successfully created harmonized multi-modal datasets across eight university hospitals for predicting outcomes after minimally invasive heart valve replacement.
• Demonstrated concurrent filtering capabilities for diverse data types, including imaging, waveforms, annotations, and metadata.
• Streamlined dataset creation for federated training, overcoming challenges of data heterogeneity.

Conclusions:

• DICOM-structured reports effectively bridge imaging and information systems, enabling concurrent querying of arbitrary data types.
• The developed platform facilitates the creation of meaningful cohorts for multi-centric data analysis in federated learning.
• The open platform and templates are available to promote reproducible multi-modal dataset creation for medical research.