Leveraging multi-modal foundation models for analysing spatial multi-omic and histopathology data.

Area of Science:

• Computational Biology
• Pathology
• Artificial Intelligence

Background:

• Pathology foundation models excel with histopathology images, while spatial multi-omic technologies provide high-resolution gene/protein expression data.
• Integrating these complementary data types remains a challenge for existing computational models.

Purpose of the Study:

• To present spEMO, a computational framework designed to unify embeddings from pathology foundation models and large language models for spatial multi-omic analysis.
• To demonstrate the superiority of multi-modal representations over single-modality models in spatial biology and pathology tasks.

Main Methods:

• Developed spEMO, a framework leveraging multi-modal representations by integrating pathology foundation models and large language models.
• Introduced a new benchmark task, multi-modal alignment, to assess pathology foundation models' ability to retrieve complementary information.

Main Results:

• spEMO surpassed single-modality models in diverse downstream tasks, including spatial domain identification, spot-type classification, and whole-slide disease prediction.
• The framework demonstrated strength in biological discovery and clinical applications, including automated medical reporting and multicellular interaction inference.
• Evaluated the effectiveness of pathology foundation models in retrieving complementary information using the new multi-modal alignment benchmark.

Conclusions:

• spEMO represents a significant advancement towards holistic, interpretable, and generalizable AI for spatial biology and pathology.
• The framework effectively integrates diverse data types, offering valuable insights for both research and clinical practice.