In-context learning enables multimodal large language models to classify cancer pathology images

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computational Pathology

Background:

• Medical image classification typically requires extensive, task-specific datasets for training deep learning models, a process that is computationally intensive and technically challenging.
• In-context learning, a method where models learn from prompts without parameter updates, is established in natural language processing but underexplored in medical image analysis.
• The scarcity of annotated data in specialized medical fields poses a significant barrier to developing effective AI diagnostic tools.

Purpose of the Study:

• To systematically evaluate the efficacy of Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) for cancer image processing using in-context learning.
• To assess GPT-4V's performance on three critical histopathology tasks: colorectal cancer tissue subtyping, colon polyp subtyping, and breast tumor detection in lymph node sections.
• To determine if in-context learning with large vision language models can serve as a viable alternative to traditional deep learning approaches in medical image analysis.

Main Methods:

• Evaluation of GPT-4V's in-context learning capabilities on three distinct cancer histopathology datasets.
• Comparison of GPT-4V's performance against specialized neural networks trained on the same tasks.
• Assessment of the number of samples required for effective in-context learning in medical image classification.

Main Results:

• In-context learning with GPT-4V demonstrated performance comparable to, or exceeding, specialized deep learning models across all evaluated histopathology tasks.
• GPT-4V achieved these results with a minimal number of samples, highlighting the efficiency of in-context learning.
• The study confirmed that generalist AI models, trained on non-domain specific data, can be effectively applied to medical image processing tasks out-of-the-box.

Conclusions:

• Large vision language models, such as GPT-4V, can be successfully applied to medical image processing tasks, specifically in histopathology, using in-context learning.
• In-context learning offers a powerful, data-efficient alternative to traditional model training for medical image analysis, democratizing AI accessibility.
• This approach holds significant promise for medical experts, particularly in resource-limited settings or areas with scarce annotated data, enabling broader adoption of AI tools.