End-to-end multimodal pathology foundation model with clinical dialogue

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Digital pathology

Background:

• Foundation models are advancing computational pathology, but clinical utility is limited.
• Current models often focus on image patches, not whole-slide analysis.
• Bridging histomorphology and diagnostic reasoning is crucial for clinical application.

Purpose of the Study:

• Introduce PRISM2, a multimodal slide-level foundation model.
• Align histomorphology with diagnostic reasoning using clinical dialogue supervision.
• Develop generalizable pathology representations for prompt-based inference and downstream tasks.

Main Methods:

• Trained PRISM2 on 2.3 million whole-slide images and 14 million QA pairs from 700,000 pathology reports.
• Utilized clinical dialogue supervision for pretraining.
• Evaluated model performance using prompt-based inference and transferable embeddings.

Main Results:

• PRISM2 achieved or exceeded balanced accuracy of clinical-grade products for cancer detection in prostate, breast, and lymph node.
• PRISM2 embeddings outperformed previous foundation models across diagnostic, biomarker, and survival benchmarks.
• Task-specific fine-tuning on survival prediction surpassed training from scratch.

Conclusions:

• Language-supervised pretraining offers a scalable, clinically grounded signal for pathology representations.
• PRISM2 effectively bridges human diagnostic reasoning and foundation model capabilities.
• This work advances the clinical utility of foundation models in pathology.