Towards generalizable pathology reports via a multimodal LLM with the multicenter in-context learning

Pathology report generation has received increasing attention in recent years. However, existing pathology report generation methods still face two main limitations: (1) these methods utilize image-report datasets where some report contents are irrelevant to the given whole slide image (WSI), making it unreliable to train an AI model to generate such content; (2) existing methods trained on data from internal hospitals struggle to generalize to external hospitals due to significant and inevitable textual discrepancies across multiple centers. To address these challenges, we first introduce a multicenter microscopic findings (MMF) dataset, which consists of WSIs and corresponding reports of lung adenocarcinoma from multiple hospitals. The MMF dataset is designed to enable the model to generate relevant report contents and evaluate its generalization to external hospitals. Second, we propose a novel Multicenter In-Context Learning (MICL) method that effectively generalizes the model to external hospitals without the need for fine-tuning. Third, we propose a new WSI-MLLM that incorporates gigabyte-sized WSIs and their image pyramid structure into MLLMs for the first time. Experiments demonstrate that WSI-MLLM significantly outperforms existing pathology report generation methods, and MICL effectively accommodates multicenter discrepancies, achieving consistent performances across hospitals and improving BLEU-4 scores by up to 26.04%. These findings underscore the effectiveness of our WSI-MLLM and MICL in generating high-quality, generalizable pathology reports. This project is available at https://huggingface.co/datasets/yili7eli/MMF.