A vision-language foundation model for precision oncology.

Area of Science:

• Artificial intelligence in medicine
• Computational pathology
• Medical informatics

Background:

• Clinical decision-making relies on integrating diverse data, such as clinical notes and pathology.
• Developing effective artificial intelligence (AI) models is challenging due to the scarcity of well-annotated multimodal clinical datasets.
• Existing AI approaches struggle to fully leverage the complementary information present in both imaging and textual clinical data.

Purpose of the Study:

• To develop a novel vision-language foundation model, Multimodal transformer with Unified maSKed modeling (MUSK), capable of integrating large-scale, unlabelled, unpaired pathology images and text data.
• To pretrain MUSK on extensive pathology image and text datasets to enable effective alignment of visual and language features.
• To evaluate MUSK's performance across a broad spectrum of clinical applications with minimal or no further training.

Main Methods:

• Developed MUSK, a vision-language foundation model utilizing unified masked modeling for pretraining on 50 million pathology images and one billion text tokens.
• Further pretrained MUSK on one million pathology image-text pairs to align vision and language modalities.
• Tested MUSK on 23 patch-level and slide-level benchmarks, including retrieval, visual question answering, classification, and outcome prediction tasks.

Main Results:

• MUSK demonstrated superior performance across 23 benchmarks, excelling in image-to-text and text-to-image retrieval, visual question answering, and image classification.
• Achieved strong results in molecular biomarker prediction and outcome prediction, including melanoma relapse, pan-cancer prognosis, and immunotherapy response in lung and gastro-oesophageal cancers.
• Effectively integrated information from pathology images and clinical reports, showcasing its potential for improving cancer diagnosis and therapy.

Conclusions:

• MUSK, a pretrained vision-language foundation model, successfully integrates multimodal pathology data (images and text) for advanced clinical applications.
• The model shows significant promise in enhancing diagnostic accuracy and precision in cancer therapy through improved outcome prediction.
• MUSK's ability to leverage large-scale, unlabelled, unpaired data represents a significant advancement in applying AI to clinical decision-making.