A visual-language foundation model for pathology image analysis using medical Twitter.

Area of Science:

• Computational pathology
• Medical artificial intelligence
• Data curation

Background:

• Annotated medical images are scarce, hindering computational research and education.
• Clinicians share de-identified images and knowledge on public platforms like medical Twitter.
• Crowdsourcing offers a potential solution for curating large-scale medical datasets.

Purpose of the Study:

• To curate a large, annotated dataset of pathology images from public sources.
• To develop a multimodal artificial intelligence model (PLIP) for pathology image analysis.
• To demonstrate the utility of the curated dataset and the developed AI model in enhancing medical diagnosis and knowledge sharing.

Main Methods:

• Harnessed public forums (e.g., medical Twitter) to curate OpenPath, a dataset of 208,414 pathology images with natural language descriptions.
• Developed pathology language-image pretraining (PLIP), a multimodal AI model trained on the OpenPath dataset.
• Evaluated PLIP's performance on external datasets for zero-shot classification and compared it with existing models.

Main Results:

• PLIP achieved state-of-the-art zero-shot classification F1 scores (0.565-0.832) on external pathology image datasets, significantly outperforming previous models (0.030-0.481).
• Training a supervised classifier on PLIP embeddings resulted in a 2.5% improvement in F1 scores compared to other supervised models.
• PLIP demonstrated effective case retrieval using either image or natural language queries, facilitating knowledge sharing.

Conclusions:

• Publicly shared medical information on crowd platforms is a valuable resource for developing medical AI.
• The OpenPath dataset and PLIP model advance computational pathology research, education, and diagnostic capabilities.
• This approach highlights the potential of leveraging crowdsourced data for creating robust medical artificial intelligence tools.