IPATH: A Large-Scale Pathology Image-Text Dataset from Instagram for Vision-Language Model Training

Area of Science:

• Artificial Intelligence
• Computational Pathology
• Medical Informatics

Background:

• Artificial intelligence (AI) can detect subtle patterns in pathology images, enhancing diagnostic accuracy.
• A significant barrier to AI development in pathology is the scarcity of large, publicly available, annotated image datasets.
• Social media platforms offer a potential, yet unexplored, source for medical image data.

Purpose of the Study:

• To curate a novel dataset of pathology images from Instagram for AI model training.
• To develop and evaluate a multimodal AI model (IP-CLIP) using this curated dataset.
• To demonstrate the utility of social media data in advancing AI for medical image analysis.

Main Methods:

• Curated the IPATH dataset, comprising 45,609 pathology image-text pairs from Instagram, using automated classifiers, large language models, and manual filtering for quality control.
• Developed IP-CLIP by fine-tuning a pre-trained CLIP model on the IPATH dataset.
• Evaluated IP-CLIP's performance on seven external histopathology datasets using zero-shot classification and linear probing, and assessed image-text alignment via retrieval on a held-out subset.

Main Results:

• IP-CLIP consistently outperformed the original CLIP model on external datasets in zero-shot classification and linear probing tasks.
• IP-CLIP achieved performance comparable to or exceeding state-of-the-art pathology vision-language models, despite training on a smaller dataset.
• IP-CLIP demonstrated superior image-text alignment compared to CLIP and specialized models in retrieval tasks.

Conclusions:

• The IPATH dataset, sourced from Instagram, is a valuable resource for developing AI in medical image classification.
• Leveraging social media data can effectively address the scarcity of annotated medical images for AI training.
• The developed IP-CLIP model shows significant potential for enhancing diagnostic accuracy and decision support in pathology.