Multimodal learning for scalable representation of high-dimensional medical data.

Area of Science:

• Computational biology
• Medical informatics
• Artificial intelligence in medicine

Background:

• Integrating multimodal healthcare data like digital pathology whole slide images (WSIs) and genomic sequencing is crucial for precision medicine.
• Current diagnostic models often fail to leverage cross-modal interactions, limiting clinical insights.
• Heterogeneity of data modalities and the need for scalable, interpretable frameworks pose significant challenges.

Purpose of the Study:

• To introduce MarbliX, a self-supervised framework for embedding WSIs and immunogenomic profiles into compact binary codes.
• To capture high-resolution patient similarity in a unified latent space for efficient case retrieval and reasoning.
• To evaluate MarbliX's performance in lung and kidney cancer diagnostics.

Main Methods:

• Developed MarbliX, a self-supervised framework utilizing multimodal association and retrieval.
• Employed a triplet contrastive objective to learn embeddings from WSIs and immunogenomic data.
• Generated compact, scalable binary codes (monograms) for unified patient representation.

Main Results:

• MarbliX achieved 85%-89% performance in lung cancer, surpassing unimodal histopathology (69%-71%) and immunogenomics (73%-76%).
• In kidney cancer, real-valued monograms achieved F1 scores of 80%-83% and accuracy of 87%-90%.
• Binary monograms in kidney cancer showed slightly lower performance (F1: 78%-82%).

Conclusions:

• MarbliX effectively integrates diverse data modalities for enhanced patient similarity analysis.
• The framework demonstrates superior performance in cancer diagnostics compared to unimodal approaches.
• MarbliX offers a scalable and interpretable solution for leveraging multimodal data in precision medicine.