Quilt-1M: One Million Image-Text Pairs for Histopathology.

Area of Science:

• Digital Pathology
• Computer Vision
• Medical Informatics

Background:

• Multi-modal learning has advanced rapidly due to abundant online image and text data.
• Histopathology lacks similar large-scale, analogous datasets, hindering progress in representation learning.
• Expert-curated educational histopathology videos on YouTube present an untapped resource.

Purpose of the Study:

• To curate a large-scale vision-language dataset for histopathology to enable representation learning.
• To address the data scarcity in medical imaging, specifically in histopathology.
• To develop and evaluate a model for histopathology image classification and cross-modal retrieval.

Main Methods:

• Automated curation of histopathology videos from YouTube to create the Quilt dataset (768,826 image-text pairs).
• Integration of Quilt with other data sources to form Quilt-1M (1 million image-text samples), the largest dataset of its kind.
• Fine-tuning a pre-trained CLIP model on the Quilt-1M dataset.

Main Results:

• The Quilt-1M dataset significantly surpasses existing histopathology datasets in scale.
• The fine-tuned CLIP model achieved state-of-the-art performance on zero-shot and linear probing tasks.
• The model demonstrated superior accuracy in classifying histopathology images across diverse datasets and improved cross-modal retrieval capabilities.

Conclusions:

• Quilt-1M is a valuable resource for advancing histopathology representation learning.
• The developed model shows strong potential for applications in digital pathology and medical image analysis.
• Leveraging online video platforms can effectively address data limitations in specialized medical fields.