Towards generalizable pathology reports via a multimodal LLM with the multicenter in-context learning

Area of Science:

• Digital pathology
• Artificial intelligence in medicine
• Natural language generation

Background:

• Current AI methods for pathology report generation face challenges with irrelevant data and poor generalization across hospitals.
• Existing models struggle with textual discrepancies inherent in multicenter datasets, limiting their real-world applicability.

Purpose of the Study:

• To develop a robust AI system for generating accurate pathology reports from whole slide images (WSIs).
• To address the limitations of existing methods by improving data relevance and enhancing generalization across multiple institutions.
• To introduce a novel approach for multicenter pathology report generation that overcomes data discrepancies.

Main Methods:

• Introduction of a multicenter microscopic findings (MMF) dataset comprising WSIs and reports for lung adenocarcinoma from various hospitals.
• Proposal of a novel Multicenter In-Context Learning (MICL) method for effective model generalization without fine-tuning.
• Development of a new Whole Slide Image-Multimodal Large Language Model (WSI-MLLM) integrating gigabyte-sized WSIs and their pyramid structure into MLLMs.

Main Results:

• The proposed WSI-MLLM significantly outperforms existing pathology report generation methods.
• The MICL method effectively handles multicenter discrepancies, achieving consistent performance across different hospitals.
• Improvements in BLEU-4 scores reached up to 26.04%, demonstrating enhanced report generation quality.

Conclusions:

• The WSI-MLLM and MICL method represent a significant advancement in automated pathology report generation.
• The developed approach enhances the quality and generalizability of AI-generated pathology reports, particularly in multicenter settings.
• This work provides a valuable resource (MMF dataset) and methodology for future research in digital pathology and AI.