MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Observation and Localization in CT Images.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Oncology

Background:

• Pancreatic cancer is a deadly malignancy often diagnosed late due to subtle CT findings.
• Current AI models struggle with accurate pancreas and tumor detection in computed tomography (CT) images.

Purpose of the Study:

• To introduce MiniGPT-Pancreas, a lightweight multimodal large language model (MLLM) for interpreting natural language queries and CT images.
• To achieve precise bounding-box predictions for the pancreas and associated tumors.
• To evaluate MiniGPT-Pancreas's performance against other leading MLLMs and its potential as a training tool for surgeons.

Main Methods:

• Cascaded fine-tuning of the MiniGPTv2 MLLM using NIH and MSD pancreas datasets.
• Evaluation of pancreas detection using intersection over Union (IoU) on NIH and MSD datasets.
• Assessment of tumor detection and localization accuracy, precision, recall, and F1 scores on the MSD dataset.
• Multi-organ detection evaluation on the AbdomenCT-1k dataset.
• Expert surgeon rating of MiniGPT-Pancreas as a training tool.

Main Results:

• MiniGPT-Pancreas achieved an average IoU of 0.57 for pancreas detection, outperforming baseline and other MLLMs.
• Tumor observation metrics (accuracy, precision, recall, F1) reached approximately 0.87 on the MSD dataset.
• Tumor localization IoU was 0.28, surpassing UMIT but lower than GLM-4.1V-9B-Base.
• On multi-organ detection, MiniGPT-Pancreas showed superior performance across all organs compared to GLM-4.1V-9B-Base and UMIT.
• Expert surgeons highly rated MiniGPT-Pancreas for verification and training of young specialists (4.5/5.0).

Conclusions:

• Compact MLLMs like MiniGPT-Pancreas can achieve performance comparable to specialized vision networks in pancreas imaging.
• MiniGPT-Pancreas offers an intuitive, language-driven approach for AI-assisted radiology.
• The model demonstrates significant potential as a valuable training tool for surgeons in pancreas imaging interpretation.