Dissecting and directing pathology foundation models

Area of Science:

• Computational Pathology
• Artificial Intelligence in Medicine
• Digital Pathology

Background:

• Foundation models (FMs) are crucial in digital pathology, encoding histology images into embeddings for diagnostics and outcome prediction.
• The 'black box' nature of these embeddings limits clinical trust and scientific discovery potential.
• Existing FM embeddings lack transparency, hindering their full clinical translation and utility.

Purpose of the Study:

• To introduce PICASSO (Pathology Image Concept Atlas built via SparSe dictiOnary learning), a framework for interpretable and controllable pathology FMs.
• To decompose FM embeddings into human-interpretable visual concepts.
• To enable auditing, discovery, and controlled manipulation of pathology FM representations.

Main Methods:

• Developed PICASSO, a framework utilizing sparse autoencoder for decomposing FM embeddings into visual concepts.
• Trained PICASSO on over 120 million tissue patches across 32 cancer types to create a pan-cancer atlas of histomorphological concepts.
• Integrated PICASSO-derived concepts with spatial transcriptomics and explored concept manipulation for counterfactual analysis.

Main Results:

• PICASSO successfully decomposes FM embeddings into interpretable concepts, creating the first pan-cancer atlas of histomorphological concepts.
• Demonstrated PICASSO's utility in auditing clinical models, discovering novel biomarkers (e.g., hobnailing morphology for EGFR mutations), and linking morphology to gene expression.
• Showcased PICASSO's ability to mitigate technical artifacts and enable controlled manipulation of embeddings for therapeutic analysis, such as survival outcome prediction.

Conclusions:

• PICASSO provides a principled framework to transform opaque pathology FMs into interpretable platforms for mechanistic insight and discovery.
• The framework enhances trustworthiness and utility of FMs in digital pathology for both clinical validation and scientific exploration.
• PICASSO facilitates the discovery of new biological insights and enables precise control over FM representations for advanced research applications.