Multi-Modal Foundation Models for Computational Pathology: A Survey

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Digital pathology

Background:

• Foundation models are advancing computational pathology (CPath) with scalable and generalizable histopathological image analysis.
• Recent progress emphasizes multi-modal foundation models integrating visual data with textual reports, domain knowledge, and molecular profiles.

Purpose of the Study:

• To provide a comprehensive review of multi-modal foundation models in CPath.
• To focus on models utilizing hematoxylin and eosin (H&E) stained whole slide images (WSIs) and tile-level data.
• To categorize existing models, datasets, tasks, and strategies, identifying future research avenues.

Main Methods:

• Categorization of 34 state-of-the-art multi-modal foundation models into vision-language, vision-knowledge graph, and vision-gene expression paradigms.
• Further classification of vision-language models into non-LLM-based and LLM-based approaches.
• Analysis and grouping of 30 pathology-focused multi-modal datasets into image-text, instruction, and image-other modality pairs.

Main Results:

• Identified three primary multi-modal foundation model paradigms: vision-language, vision-knowledge graph, and vision-gene expression.
• Cataloged 34 distinct multi-modal foundation models and 30 relevant datasets.
• Developed a taxonomy of downstream tasks and analyzed training/evaluation strategies.

Conclusions:

• Multi-modal foundation models represent a significant advancement in computational pathology.
• This survey offers a structured overview and resource for researchers in AI and pathology.
• Key challenges and future directions are highlighted to guide further development in the field.